[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487864429 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003225] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008493] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010024] pid_max: default: 32768 minimum: 301 [ 0.011171] LSM: Security Framework initializing [ 0.012080] Yama: becoming mindful. [ 0.013057] SELinux: Initializing. [ 0.014105] *** VALIDATE selinux *** [ 0.022654] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027476] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028194] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030047] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031142] *** VALIDATE tmpfs *** [ 0.032536] *** VALIDATE proc *** [ 0.033320] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036316] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039040] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.043129] debug: unmapping init [mem 0xffffffffbb059000-0xffffffffbb060fff] [ 0.046173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047756] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052028] ... max period: 00007fffffffffff [ 0.053019] ... fixed-purpose events: 3 [ 0.054016] ... event mask: 000000070000000f [ 0.056287] rcu: Hierarchical SRCU implementation. [ 0.058683] smp: Bringing up secondary CPUs ... [ 0.059671] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.063764] smp: Brought up 1 node, 4 CPUs [ 0.065030] smpboot: Max logical packages: 1 [ 0.066027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.216024] node 0 deferred pages initialised in 148ms [ 0.219363] devtmpfs: initialized [ 0.221324] x86/mm: Memory block size: 128MB [ 0.225510] gcov: version magic: 0x41383552 [ 0.229302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.234172] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.237418] pinctrl core: initialized pinctrl subsystem [ 0.240233] [ 0.240984] ************************************************************* [ 0.245017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248012] ** ** [ 0.251012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.254014] ** ** [ 0.257017] ** This means that this kernel is built to expose internal ** [ 0.259012] ** IOMMU data structures, which may compromise security on ** [ 0.262013] ** your system. ** [ 0.265018] ** ** [ 0.268067] ** If you see this message and you are not debugging the ** [ 0.271012] ** kernel, report this immediately to your vendor! ** [ 0.274014] ** ** [ 0.277014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280014] ************************************************************* [ 0.283141] NET: Registered protocol family 16 [ 0.285484] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.289066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.292083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297025] cpuidle: using governor menu [ 0.299007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300530] PCI: Using configuration type 1 for base access [ 0.303014] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313021] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.316035] cryptd: max_cpu_qlen set to 1000 [ 0.319332] ACPI: Added _OSI(Module Device) [ 0.321020] ACPI: Added _OSI(Processor Device) [ 0.323018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.324017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329137] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335636] ACPI: Interpreter enabled [ 0.338076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.340029] ACPI: Using IOAPIC for interrupt routing [ 0.342140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.347464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.358162] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.361048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.364024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.368096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.373419] acpiphp: Slot [2] registered [ 0.375139] acpiphp: Slot [5] registered [ 0.377140] acpiphp: Slot [6] registered [ 0.379148] acpiphp: Slot [7] registered [ 0.381081] acpiphp: Slot [8] registered [ 0.383120] acpiphp: Slot [9] registered [ 0.384117] acpiphp: Slot [10] registered [ 0.386117] acpiphp: Slot [3] registered [ 0.387090] acpiphp: Slot [4] registered [ 0.388071] acpiphp: Slot [11] registered [ 0.389123] acpiphp: Slot [12] registered [ 0.391177] acpiphp: Slot [13] registered [ 0.393114] acpiphp: Slot [14] registered [ 0.395120] acpiphp: Slot [15] registered [ 0.396089] acpiphp: Slot [16] registered [ 0.397095] acpiphp: Slot [17] registered [ 0.399102] acpiphp: Slot [18] registered [ 0.400135] acpiphp: Slot [19] registered [ 0.401086] acpiphp: Slot [20] registered [ 0.402074] acpiphp: Slot [21] registered [ 0.403025] acpiphp: Slot [22] registered [ 0.404084] acpiphp: Slot [23] registered [ 0.405087] acpiphp: Slot [24] registered [ 0.406107] acpiphp: Slot [25] registered [ 0.407097] acpiphp: Slot [26] registered [ 0.408077] acpiphp: Slot [27] registered [ 0.409081] acpiphp: Slot [28] registered [ 0.411086] acpiphp: Slot [29] registered [ 0.412122] acpiphp: Slot [30] registered [ 0.413110] acpiphp: Slot [31] registered [ 0.414082] PCI host bridge to bus 0000:00 [ 0.416032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.419027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.424022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.426028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.429047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.430154] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.433052] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.436655] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445882] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.451090] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.453017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.456026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.458016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.462529] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.465782] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.472646] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.478017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.490870] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.496015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.502961] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.509014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.516024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.535022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.543736] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.552024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.559021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.577019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.587661] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.593020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.600022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.614032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.624499] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.630017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.637019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.651017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.666618] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.673021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.678140] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.693023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.700904] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.708018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.717020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.739019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.751487] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.754416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.756363] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.759403] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.761227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.766040] iommu: Default domain type: Passthrough [ 0.768286] SCSI subsystem initialized [ 0.769144] ACPI: bus type USB registered [ 0.771185] usbcore: registered new interface driver usbfs [ 0.774110] usbcore: registered new interface driver hub [ 0.776108] usbcore: registered new device driver usb [ 0.777218] pps_core: LinuxPPS API ver. 1 registered [ 0.778013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.781075] PTP clock support registered [ 0.783133] EDAC MC: Ver: 3.0.0 [ 0.784497] PCI: Using ACPI for IRQ routing [ 0.785540] NetLabel: Initializing [ 0.787021] NetLabel: domain hash size = 128 [ 0.789014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.791109] NetLabel: unlabeled traffic allowed by default [ 0.794045] vgaarb: loaded [ 0.796284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.797017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.805405] clocksource: Switched to clocksource kvm-clock [ 0.916055] VFS: Disk quotas dquot_6.6.0 [ 0.917323] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.919437] *** VALIDATE ramfs *** [ 0.920420] *** VALIDATE hugetlbfs *** [ 0.921772] pnp: PnP ACPI init [ 0.923826] pnp: PnP ACPI: found 6 devices [ 0.942490] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.946088] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.948259] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.950368] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.952539] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.954740] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.957172] NET: Registered protocol family 2 [ 0.959380] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.964052] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967089] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.971972] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.976433] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980073] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.983596] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.986399] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989847] NET: Registered protocol family 1 [ 0.992818] RPC: Registered named UNIX socket transport module. [ 0.995225] RPC: Registered udp transport module. [ 0.997084] RPC: Registered tcp transport module. [ 0.998886] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.001267] NET: Registered protocol family 44 [ 1.003127] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.005290] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.007212] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.009327] PCI: CLS 0 bytes, default 64 [ 1.010867] Unpacking initramfs... [ 2.433759] debug: unmapping init [mem 0xffff8bb53cc54000-0xffff8bb53ffbffff] [ 2.437410] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.441669] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.444374] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.940435] Initialise system trusted keyrings [ 2.942049] Key type blacklist registered [ 2.944266] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.953663] zbud: loaded [ 2.956956] *** VALIDATE nfs *** [ 2.957818] *** VALIDATE nfs4 *** [ 2.959537] pstore: using deflate compression [ 2.962925] Platform Keyring initialized [ 3.070137] NET: Registered protocol family 38 [ 3.071783] Key type asymmetric registered [ 3.073513] Asymmetric key parser 'x509' registered [ 3.075470] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.078736] io scheduler mq-deadline registered [ 3.080794] io scheduler kyber registered [ 3.082054] io scheduler bfq registered [ 3.083786] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.088094] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.091187] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.094245] ACPI: Power Button [PWRF] [ 3.099732] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.107931] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.122294] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.128525] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.143229] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.172753] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.201043] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.205599] Non-volatile memory driver v1.3 [ 3.207758] Linux agpgart interface v0.103 [ 3.239626] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.241980] vda: detected capacity change from 0 to 74702848 [ 3.255215] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.258172] vdb: detected capacity change from 0 to 1073741824 [ 3.279304] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.282065] vdc: detected capacity change from 0 to 2621440000 [ 3.296327] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.298903] vdd: detected capacity change from 0 to 2621440000 [ 3.313216] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.315213] vde: detected capacity change from 0 to 4294967296 [ 3.330022] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.333271] vdf: detected capacity change from 0 to 4294967296 [ 3.339441] libphy: Fixed MDIO Bus: probed [ 3.350962] usbcore: registered new interface driver usbserial_generic [ 3.353142] usbserial: USB Serial support registered for generic [ 3.354978] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.359146] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.360441] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.362451] mousedev: PS/2 mouse device common for all mice [ 3.364909] rtc_cmos 00:05: RTC can wake from S4 [ 3.367402] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.367459] rtc_cmos 00:05: registered as rtc0 [ 3.371554] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.374335] intel_pstate: CPU model not supported [ 3.377720] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.379174] hid: raw HID events driver (C) Jiri Kosina [ 3.382729] usbcore: registered new interface driver usbhid [ 3.385024] usbhid: USB HID core driver [ 3.387128] drop_monitor: Initializing network drop monitor service [ 3.387226] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.389561] Initializing XFRM netlink socket [ 3.394909] NET: Registered protocol family 10 [ 3.398189] Segment Routing with IPv6 [ 3.399602] NET: Registered protocol family 17 [ 3.401152] mpls_gso: MPLS GSO support [ 3.406934] RAS: Correctable Errors collector initialized. [ 3.409325] AVX version of gcm_enc/dec engaged. [ 3.411017] AES CTR mode by8 optimization enabled [ 3.482991] sched_clock: Marking stable (3482966427, 0)->(4400896951, -917930524) [ 3.487317] registered taskstats version 1 [ 3.489973] Loading compiled-in X.509 certificates [ 3.492923] zswap: loaded using pool lzo/zbud [ 3.519406] Key type big_key registered [ 3.535441] Key type encrypted registered [ 3.537316] ima: No TPM chip found, activating TPM-bypass! [ 3.539439] ima: Allocated hash algorithm: sha1 [ 3.541435] ima: No architecture policies found [ 3.543262] evm: Initialising EVM extended attributes: [ 3.545133] evm: security.selinux [ 3.546292] evm: security.ima [ 3.547460] evm: security.capability [ 3.548682] evm: HMAC attrs: 0x1 [ 3.550982] rtc_cmos 00:05: setting system clock to 2026-08-15 22:01:01 UTC (1786831261) [ 3.557821] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 3.561430] debug: unmapping init [mem 0xffffffffbad82000-0xffffffffbb058fff] [ 3.571106] Write protecting the kernel read-only data: 28672k [ 3.574481] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 3.578408] debug: unmapping init [mem 0xffffffffb9d14000-0xffffffffb9dfffff] [ 3.621571] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.632441] systemd[1]: Detected virtualization kvm. [ 3.634916] systemd[1]: Detected architecture x86-64. [ 3.637452] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.663728] systemd[1]: No hostname configured. [ 3.665568] systemd[1]: Set hostname to . [ 3.668387] random: systemd: uninitialized urandom read (16 bytes read) [ 3.670462] systemd[1]: Initializing machine ID from random generator. [ 3.708728] random: ln: uninitialized urandom read (6 bytes read) [ 3.790685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.792922] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.797684] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.802859] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.368105] device-mapper: uevent: version 1.0.3 [ 4.369856] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 4.670163] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.077020] virtio_net virtio0 ens2: renamed from eth0 [ 5.088452] scsi host0: ata_piix [ 5.105871] scsi host1: ata_piix [ 5.107642] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.110330] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.248731] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.054866] random: crng init done [ 10.056034] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.256369] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.314744] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.371250] SELinux: Disabled at runtime. [ 11.436520] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.446072] systemd[1]: Detected virtualization kvm. [ 11.447863] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.932211] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.935677] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.941965] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.946227] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.949620] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.962288] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.970366] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ 12.151239] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.412546] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.712321] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.747549] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.837641] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.846963] EDAC sbridge: Ver: 1.1.2 [ 14.187683] Key type dns_resolver registered [ 14.477134] NFS: Registering the id_resolver key type [ 14.479138] Key type id_resolver registered [ 14.480288] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg335-server login: [ 40.176676] libcfs: loading out-of-tree module taints kernel. [ 40.203659] Key type ._llcrypt registered [ 40.204971] Key type .llcrypt registered [ 40.247933] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 48.844834] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 49.415458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.423118] alg: No test for adler32 (adler32-zlib) [ 50.433877] Lustre: Lustre: Build Version: 2.17.57_3_g67c4c6e [ 50.780649] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 52.400220] Key type lgssc registered [ 53.292309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.810040] hrtimer: interrupt took 3031271 ns [ 67.667911] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 103.385897] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 114.668232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 114.702342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 115.898918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 115.953248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 116.090799] Lustre: lustre-MDT0000: new disk, initializing [ 116.202489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 116.244803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 120.645729] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.473335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.605349] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 133.648740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 133.654478] Lustre: Skipped 1 previous similar message [ 133.740570] Lustre: lustre-MDT0001: new disk, initializing [ 133.794951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.822223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 133.833081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 137.406714] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 141.263321] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 152.215739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 152.449438] Lustre: lustre-OST0000: new disk, initializing [ 152.452406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 152.456615] Lustre: 8424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 152.518859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.262921] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.367722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 162.381288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 162.426297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.017216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.125387] Lustre: lustre-OST0001: new disk, initializing [ 175.128793] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 175.135974] Lustre: 9498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.188312] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.666204] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.374899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.381064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.492657] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 191.781314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.852266] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 202.266654] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing check_logdir /tmp/testlogs/ [ 206.834277] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing yml_node [ 211.569026] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 214.342375] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 217.028146] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 218.778397] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat Aug 15 18:04:33 EDT 2026 [ 236.465654] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 238.460912] Lustre: DEBUG MARKER: === replay-single: start setup 18:04:53 (1786831493) === [ 244.666388] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing check_config_client /mnt/lustre [ 263.321774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.878221] Lustre: 13305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 271.256529] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.965737] Lustre: DEBUG MARKER: === replay-single: finish setup 18:05:32 (1786831532) === [ 278.766153] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 18:05:33 (1786831533) [ 287.271230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 289.468753] Lustre: Failing over lustre-MDT0000 [ 289.680786] Lustre: server umount lustre-MDT0000 complete [ 292.330742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 292.356553] Lustre: Skipped 1 previous similar message [ 297.442086] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.454855] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 299.616038] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.568206] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.587917] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 304.745752] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.513041] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831550/real 1786831550] req@ffff8bb5b4b1a680 x1873628425728128/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 308.535259] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.853687] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.875087] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 311.477562] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 311.479672] LDISKFS-fs (dm-0): recovery complete [ 311.508273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 318.944504] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb486755500 x1873628425737600/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 318.944698] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.995166] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 319.212178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 321.121933] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 323.584255] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 324.599334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 324.659913] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 332.746122] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 334.056532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 341.414887] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 18:06:36 (1786831596) [ 343.410364] Lustre: Failing over lustre-OST0000 [ 343.612860] Lustre: server umount lustre-OST0000 complete [ 345.059804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.070936] Lustre: Skipped 2 previous similar messages [ 345.074057] LustreError: 8407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.086731] LustreError: 8407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 361.810857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 362.036677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.085671] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 363.857233] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 363.881528] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 363.891253] Lustre: Skipped 3 previous similar messages [ 368.300896] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 377.272061] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 378.962588] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 387.761754] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 18:07:23 (1786831643) [ 396.171371] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 398.403065] Lustre: Failing over lustre-MDT0000 [ 398.955254] Lustre: server umount lustre-MDT0000 complete [ 401.376833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.387754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.405551] Lustre: Skipped 1 previous similar message [ 401.422862] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.458577] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 418.272375] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831660/real 1786831660] req@ffff8bb5b8ee0380 x1873628425793664/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 418.305675] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.368476] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 420.376987] LDISKFS-fs (dm-0): recovery complete [ 420.390791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.515099] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb4877ad180 x1873628425801088/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 428.743223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.806850] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 434.148152] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 434.173391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 434.179232] Lustre: Skipped 1 previous similar message [ 434.814945] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 439.898953] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 445.027594] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 450.140642] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 455.258102] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 465.504747] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 465.534462] Lustre: Skipped 1 previous similar message [ 485.997822] Lustre: lustre-MDT0000: Denying connection for new client 709ad596-5912-4c1f-908d-abef209c31cc (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 486.025255] Lustre: Skipped 3 previous similar messages [ 504.500266] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 504.508251] Lustre: 18206:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 255b0c7e-af73-4ac3-b71f-8628cd61cc16@ [ 504.523885] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 504.532106] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 504.532118] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 504.537564] Lustre: Skipped 2 previous similar messages [ 504.593547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 504.595066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 516.545401] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 18:09:31 (1786831771) [ 523.418668] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 526.129405] Lustre: Failing over lustre-MDT0000 [ 526.310783] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.314353] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.323452] Lustre: Skipped 5 previous similar messages [ 526.337633] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 526.429670] Lustre: server umount lustre-MDT0000 complete [ 547.280111] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831789/real 1786831789] req@ffff8bb48572e300 x1873628425854080/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 547.320608] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.197501] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 549.200861] LDISKFS-fs (dm-0): recovery complete [ 549.208156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.783135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.507622] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 563.173317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 563.178640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 563.582274] Lustre: lustre-MDT0000: Denying connection for new client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 563.606628] Lustre: Skipped 3 previous similar messages [ 630.367918] Lustre: lustre-MDT0000: Denying connection for new client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 630.399663] Lustre: Skipped 12 previous similar messages [ 633.500681] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 633.505856] Lustre: 19934:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 709ad596-5912-4c1f-908d-abef209c31cc@ [ 633.524882] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 633.545570] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 633.545787] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 633.586524] Lustre: Skipped 2 previous similar messages [ 633.637490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 633.641161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 646.135136] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 18:11:41 (1786831901) [ 652.869359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 654.147429] Lustre: Failing over lustre-MDT0000 [ 654.326564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.334378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.343824] Lustre: Skipped 1 previous similar message [ 654.352396] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.366153] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 654.381454] Lustre: server umount lustre-MDT0000 complete [ 672.224089] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831913/real 1786831913] req@ffff8bb5b4b1b800 x1873628425910528/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 672.270626] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.413408] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 675.417452] LDISKFS-fs (dm-0): recovery complete [ 675.426959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.478525] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c4da5e [ 682.490393] LustreError: 21639:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.135@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff8bb5b2394900) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 682.500454] Lustre: MGC192.168.203.135@tcp: Connection restored to 0@lo (at 0@lo) [ 682.585838] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 682.675215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.712865] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 684.391647] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 686.398574] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 688.165545] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 688.197178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 688.197372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 694.436183] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 696.155994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 703.731882] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 18:12:39 (1786831959) [ 710.653786] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 712.661875] Lustre: Failing over lustre-MDT0000 [ 712.895405] Lustre: server umount lustre-MDT0000 complete [ 713.697247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 713.703576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.727208] Lustre: Skipped 6 previous similar messages [ 729.059669] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831971/real 1786831971] req@ffff8bb5aef61f80 x1873628425943168/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 729.110971] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 735.024365] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 735.026593] LDISKFS-fs (dm-0): recovery complete [ 735.032585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 739.298890] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb487c83100 x1873628425950592/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 739.593453] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 739.941362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 743.521675] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 744.947755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 744.962445] Lustre: Skipped 4 previous similar messages [ 745.054303] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 745.103740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 745.107236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 752.822652] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 754.404175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 763.369729] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 18:13:38 (1786832018) [ 770.860684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 772.555637] Lustre: Failing over lustre-MDT0000 [ 772.824993] Lustre: server umount lustre-MDT0000 complete [ 775.650935] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.665129] Lustre: Skipped 2 previous similar messages [ 792.038437] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832033/real 1786832033] req@ffff8bb5c1bbad80 x1873628425977728/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 792.096022] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 795.540262] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 795.546099] LDISKFS-fs (dm-0): recovery complete [ 795.570059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.273213] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5b4045180 x1873628425986304/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 802.658353] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 802.708046] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 807.441569] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 808.042736] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 808.095905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 808.096899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 817.169280] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 819.249154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 828.534120] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 18:14:43 (1786832083) [ 836.382232] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 838.503326] Lustre: Failing over lustre-MDT0000 [ 838.646128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.654918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.675880] Lustre: Skipped 4 previous similar messages [ 838.697657] Lustre: Skipped 3 previous similar messages [ 839.026205] Lustre: server umount lustre-MDT0000 complete [ 860.129350] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832101/real 1786832101] req@ffff8bb5b8ee2680 x1873628426015104/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 860.175660] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.291785] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 861.296313] LDISKFS-fs (dm-0): recovery complete [ 861.315289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.372336] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5a93c0700 x1873628426024192/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 870.584677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.591288] Lustre: Skipped 2 previous similar messages [ 870.632773] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 872.140515] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 874.799103] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 876.021245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 876.033627] Lustre: Skipped 7 previous similar messages [ 876.228334] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 876.334672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 876.335079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 886.448693] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 888.794907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 898.146720] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 18:15:53 (1786832153) [ 905.372354] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 907.204591] Lustre: Failing over lustre-MDT0000 [ 907.239477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 907.249685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.264491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.365073] Lustre: server umount lustre-MDT0000 complete [ 911.843368] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 911.869454] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 138 previous similar messages [ 927.200727] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832169/real 1786832169] req@ffff8bb5b7594700 x1873628426052864/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 927.267690] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.175946] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 933.178526] LDISKFS-fs (dm-0): recovery complete [ 933.185624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.674150] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 938.594172] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 941.322266] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 943.140098] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 943.220587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 943.223146] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 951.709581] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.657250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 962.733406] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 18:16:58 (1786832218) [ 963.699968] Lustre: *** cfs_fail_loc=13b, val=315*** [ 963.705046] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 963.709485] LustreError: 6493:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb588c57b80 x1873628415281152/t38654705666(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:157/0 lens 392/456 e 0 to 0 dl 1786832232 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 971.939054] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 973.721537] Lustre: Failing over lustre-MDT0000 [ 973.820953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 973.825481] Lustre: Skipped 3 previous similar messages [ 973.826126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 974.246310] Lustre: server umount lustre-MDT0000 complete [ 994.259046] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832236/real 1786832236] req@ffff8bb5b8ee1500 x1873628426089472/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 994.290991] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.965648] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 995.967859] LDISKFS-fs (dm-0): recovery complete [ 995.981837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.523056] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5b7675f80 x1873628426097152/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1004.790132] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1008.657104] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1010.162985] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1010.168257] Lustre: Skipped 7 previous similar messages [ 1010.249781] Lustre: 6493:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5aefb6300 x1873628415281152/t38654705666(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:204/0 lens 392/456 e 0 to 0 dl 1786832279 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1010.303817] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1010.310342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1017.007601] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1018.785166] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1027.453768] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 18:18:02 (1786832282) [ 1034.751996] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1036.634367] Lustre: Failing over lustre-MDT0000 [ 1037.049257] Lustre: server umount lustre-MDT0000 complete [ 1040.871663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1040.871764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.883755] Lustre: Skipped 8 previous similar messages [ 1060.000158] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1060.010635] LDISKFS-fs (dm-0): recovery complete [ 1060.026837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.468303] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c4f936 [ 1066.824687] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1068.337261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1068.341849] Lustre: Skipped 1 previous similar message [ 1070.982249] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1072.190308] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1072.206749] Lustre: Skipped 1 previous similar message [ 1072.275375] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1072.276711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1080.724437] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1082.076223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1089.952094] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 18:19:05 (1786832345) [ 1097.120870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1098.127859] Lustre: *** cfs_fail_loc=114, val=0*** [ 1100.924694] Lustre: Failing over lustre-MDT0000 [ 1101.147301] Lustre: server umount lustre-MDT0000 complete [ 1122.183855] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1122.194943] LDISKFS-fs (dm-0): recovery complete [ 1122.217246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.207611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.220145] Lustre: Skipped 3 previous similar messages [ 1129.281439] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1134.790202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1134.793698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1135.763296] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1145.808742] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1147.444590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1155.953273] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 18:20:11 (1786832411) [ 1164.310280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1165.367094] Lustre: *** cfs_fail_loc=128, val=0*** [ 1168.149019] Lustre: Failing over lustre-MDT0000 [ 1168.440358] Lustre: server umount lustre-MDT0000 complete [ 1186.257501] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832428/real 1786832428] req@ffff8bb5b43adc00 x1873628426196992/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1186.292350] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1186.305284] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.321814] LustreError: Skipped 2 previous similar messages [ 1190.227446] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1190.229717] LDISKFS-fs (dm-0): recovery complete [ 1190.235643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.515070] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5aefb4000 x1873628426205568/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1196.637897] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 1196.753963] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1200.181141] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1202.258653] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1202.260671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1208.352832] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1209.690996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1217.082720] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 18:21:12 (1786832472) [ 1224.294241] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1226.359765] Lustre: Failing over lustre-MDT0000 [ 1226.608032] Lustre: server umount lustre-MDT0000 complete [ 1227.749162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1247.677532] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1247.680716] LDISKFS-fs (dm-0): recovery complete [ 1247.695229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.464774] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1259.343663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1259.344326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1267.945470] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1269.356269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1277.258501] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 18:22:12 (1786832532) [ 1284.286546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1286.292957] Lustre: Failing over lustre-MDT0000 [ 1286.515378] Lustre: server umount lustre-MDT0000 complete [ 1307.452427] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1307.457025] LDISKFS-fs (dm-0): recovery complete [ 1307.470073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.296702] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb487242300 x1873628426273024/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1320.477929] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1320.944436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1320.958498] Lustre: Skipped 20 previous similar messages [ 1321.164979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1321.174075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1328.994334] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1330.407709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1338.943099] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 18:23:13 (1786832593) [ 1347.352926] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1356.493823] Lustre: Failing over lustre-MDT0000 [ 1356.742907] Lustre: server umount lustre-MDT0000 complete [ 1356.777548] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.797667] Lustre: Skipped 17 previous similar messages [ 1381.217279] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1381.220779] LDISKFS-fs (dm-0): recovery complete [ 1381.238965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.842327] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1388.855168] Lustre: Skipped 2 previous similar messages [ 1390.565921] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1390.577151] Lustre: Skipped 4 previous similar messages [ 1392.622638] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1397.674170] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 1397.681912] Lustre: Skipped 4 previous similar messages [ 1397.734191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1397.739282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1403.251778] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.694915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1428.569049] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 18:24:43 (1786832683) [ 1437.246520] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1439.362796] Lustre: Failing over lustre-MDT0000 [ 1439.816708] Lustre: server umount lustre-MDT0000 complete [ 1440.226503] LustreError: 8677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.250796] LustreError: 8677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 253 previous similar messages [ 1456.616348] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832698/real 1786832698] req@ffff8bb4875cb480 x1873628426407552/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1456.651621] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1456.664872] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.680098] LustreError: Skipped 3 previous similar messages [ 1462.077382] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1462.079557] LDISKFS-fs (dm-0): recovery complete [ 1462.087142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.850045] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5be4c0700 x1873628426416128/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1471.150735] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1472.622552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1472.629406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1479.674836] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1481.214404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1490.453198] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 18:25:45 (1786832745) [ 1497.453568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1499.354607] Lustre: Failing over lustre-MDT0000 [ 1499.630255] Lustre: server umount lustre-MDT0000 complete [ 1503.201377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.373406] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1520.377079] LDISKFS-fs (dm-0): recovery complete [ 1520.389401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.259734] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1535.577689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1535.577992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1544.922776] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1546.465315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1555.814494] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 18:26:50 (1786832810) [ 1564.616559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1566.688590] Lustre: Failing over lustre-MDT0000 [ 1566.851164] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 1567.075618] Lustre: server umount lustre-MDT0000 complete [ 1571.299548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1588.832852] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1588.834596] LDISKFS-fs (dm-0): recovery complete [ 1588.843750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.921231] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5be634000 x1873628426498432/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1602.915661] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1603.689616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1603.691254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1613.453320] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1615.189330] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1623.543772] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 18:27:58 (1786832878) [ 1631.284450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1633.489247] Lustre: Failing over lustre-MDT0000 [ 1633.734188] Lustre: server umount lustre-MDT0000 complete [ 1658.131787] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1658.138995] LDISKFS-fs (dm-0): recovery complete [ 1658.152806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.384236] LustreError: 51015:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1660.394397] LustreError: 51015:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8bb5be7c2680 x1873628426531200/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786832918 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1660.420026] LustreError: 51015:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1661.030135] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 1661.207647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.223678] Lustre: Skipped 7 previous similar messages [ 1661.274914] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1661.287642] Lustre: Skipped 3 previous similar messages [ 1666.605132] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1666.704045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1666.707046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1678.107969] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.746916] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1690.193851] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 18:29:04 (1786832944) [ 1698.386309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1700.763618] Lustre: Failing over lustre-MDT0000 [ 1701.064372] Lustre: server umount lustre-MDT0000 complete [ 1727.220637] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1727.226617] LDISKFS-fs (dm-0): recovery complete [ 1727.247874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.481042] LustreError: 52952:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1728.489614] LustreError: 52952:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8bb5bf854e00 x1873628426572160/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786832986 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1728.508901] LustreError: 52952:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1734.767917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1734.768080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1735.141604] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1745.279782] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1746.836441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1755.805679] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 18:30:10 (1786833010) [ 1762.970908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1764.813817] Lustre: Failing over lustre-MDT0000 [ 1765.035168] Lustre: server umount lustre-MDT0000 complete [ 1765.344956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1791.102885] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1791.108112] LDISKFS-fs (dm-0): recovery complete [ 1791.132063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.979226] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c5ca8e [ 1792.099031] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 1797.074607] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1797.829931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1797.830877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1809.588436] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1811.348172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1822.540883] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 18:31:16 (1786833076) [ 1831.192204] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1833.850460] Lustre: Failing over lustre-MDT0000 [ 1834.502991] Lustre: server umount lustre-MDT0000 complete [ 1858.195851] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1858.200175] LDISKFS-fs (dm-0): recovery complete [ 1858.209174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.160980] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb592cfd500 x1873628426651520/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1864.215517] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 10 previous similar messages [ 1868.420755] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1869.813620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1869.823760] Lustre: Skipped 32 previous similar messages [ 1869.943879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1869.947461] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1877.484733] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1879.082492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1887.637555] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 18:32:22 (1786833142) [ 1894.987598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1896.759409] Lustre: Failing over lustre-MDT0000 [ 1897.015580] Lustre: server umount lustre-MDT0000 complete [ 1900.513075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.527945] Lustre: Skipped 31 previous similar messages [ 1920.923799] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1920.926358] LDISKFS-fs (dm-0): recovery complete [ 1920.934492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.288496] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1928.301334] Lustre: Skipped 7 previous similar messages [ 1930.514957] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1931.811606] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1931.818857] Lustre: Skipped 7 previous similar messages [ 1931.861310] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1931.861310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1940.095944] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1941.372107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1949.619674] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 18:33:24 (1786833204) [ 1957.679734] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1959.691225] Lustre: Failing over lustre-MDT0000 [ 1960.153031] Lustre: server umount lustre-MDT0000 complete [ 1978.849657] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833220/real 1786833220] req@ffff8bb5a924ea00 x1873628426713984/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786833236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1978.897397] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 1978.920156] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1978.953820] LustreError: Skipped 7 previous similar messages [ 1984.313079] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1984.320930] LDISKFS-fs (dm-0): recovery complete [ 1984.339859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.433458] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1993.770298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1993.772773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 2004.532857] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2006.985488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2016.655746] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 18:34:31 (1786833271) [ 2024.949434] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2026.808829] Lustre: Failing over lustre-MDT0000 [ 2027.153935] Lustre: server umount lustre-MDT0000 complete [ 2029.537592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2029.544129] LustreError: Skipped 3 previous similar messages [ 2040.936067] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.998296] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 315 previous similar messages [ 2052.358462] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2052.360562] LDISKFS-fs (dm-0): recovery complete [ 2052.385881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.737712] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2060.903555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 2060.906907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 2071.192420] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.066912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2085.108250] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 18:35:39 (1786833339) [ 2096.271804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2098.679293] Lustre: Failing over lustre-MDT0000 [ 2099.289914] Lustre: server umount lustre-MDT0000 complete [ 2120.266297] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2120.269757] LDISKFS-fs (dm-0): recovery complete [ 2120.284785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.480794] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2133.072035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2133.073118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2140.318675] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.808972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2151.251578] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 18:36:46 (1786833406) [ 2158.033456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2159.837037] Lustre: Failing over lustre-MDT0000 [ 2160.336690] Lustre: server umount lustre-MDT0000 complete [ 2182.995571] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2183.005092] LDISKFS-fs (dm-0): recovery complete [ 2183.021274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.324613] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c5ec1b [ 2190.555573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2190.571704] Lustre: Skipped 7 previous similar messages [ 2195.006648] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2196.061093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2196.062047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2204.105907] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2205.662359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2213.366308] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 18:37:48 (1786833468) [ 2220.022237] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2221.722811] Lustre: Failing over lustre-MDT0000 [ 2221.879127] Lustre: server umount lustre-MDT0000 complete [ 2243.078678] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2243.083508] LDISKFS-fs (dm-0): recovery complete [ 2243.106966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.495439] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2258.036951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2258.041296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2265.538839] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.368808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2276.223700] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 18:38:51 (1786833531) [ 2283.154203] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2285.319221] Lustre: Failing over lustre-MDT0000 [ 2285.641453] Lustre: server umount lustre-MDT0000 complete [ 2307.934788] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2307.938184] LDISKFS-fs (dm-0): recovery complete [ 2307.945956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.220661] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c5f712 [ 2314.495416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.500803] Lustre: Skipped 9 previous similar messages [ 2319.214726] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2319.985058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2319.986056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2327.991687] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.777351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2338.096975] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 18:39:53 (1786833593) [ 2345.346833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2347.415690] Lustre: Failing over lustre-MDT0000 [ 2347.909854] Lustre: server umount lustre-MDT0000 complete [ 2370.962625] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2370.967624] LDISKFS-fs (dm-0): recovery complete [ 2370.980788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.174673] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c5fdef [ 2380.491075] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2381.916508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2381.920043] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2389.719468] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.212361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2398.869694] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:40:54 (1786833654) [ 2405.268691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2406.748375] Lustre: Failing over lustre-MDT0000 [ 2406.948695] Lustre: server umount lustre-MDT0000 complete [ 2427.885177] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2427.889420] LDISKFS-fs (dm-0): recovery complete [ 2427.897805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.888932] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2439.829071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 2439.831034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 2448.305513] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2450.408394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2458.651489] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 18:41:54 (1786833714) [ 2463.993321] Lustre: 75303:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting aea50010-778e-4c91-a65b-6c2b6549a881 at adminstrative request [ 2469.148529] Lustre: Failing over lustre-MDT0000 [ 2469.382363] Lustre: server umount lustre-MDT0000 complete [ 2487.731611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.998997] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c608b5 [ 2497.004039] Lustre: MGC192.168.203.135@tcp: Connection restored to 0@lo (at 0@lo) [ 2497.006545] Lustre: Skipped 42 previous similar messages [ 2501.529615] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2502.756299] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 2502.756960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2512.906864] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2514.892774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2521.219752] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2531.478294] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2537.124178] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 18:43:12 (1786833792) [ 2538.036877] Lustre: 77331:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting aea50010-778e-4c91-a65b-6c2b6549a881 at adminstrative request [ 2545.971482] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:43:21 (1786833801) [ 2553.112566] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2554.832674] Lustre: Failing over lustre-MDT0000 [ 2555.150097] Lustre: server umount lustre-MDT0000 complete [ 2558.944655] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.951728] Lustre: Skipped 40 previous similar messages [ 2576.710111] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2576.719158] LDISKFS-fs (dm-0): recovery complete [ 2576.731702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.671141] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 2584.686331] Lustre: Skipped 1 previous similar message [ 2585.882188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2585.893881] Lustre: Skipped 9 previous similar messages [ 2589.765237] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2590.376762] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2590.398190] Lustre: Skipped 9 previous similar messages [ 2590.458411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2590.458540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2598.853113] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2600.151867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2608.145473] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:44:23 (1786833863) [ 2615.909269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2617.903536] Lustre: Failing over lustre-MDT0000 [ 2618.478847] Lustre: server umount lustre-MDT0000 complete [ 2636.192312] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833878/real 1786833878] req@ffff8bb5b4131500 x1873628427092352/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786833894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2636.217121] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2636.220885] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.225965] LustreError: Skipped 9 previous similar messages [ 2639.125254] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2639.127558] LDISKFS-fs (dm-0): recovery complete [ 2639.132688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.378046] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.394325] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 306 previous similar messages [ 2646.500758] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c613ba [ 2646.511130] LustreError: 80586:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.135@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff8bb5a1277000) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2650.849244] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2652.198823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2652.203811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2659.110844] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2660.130393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2667.426256] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:45:22 (1786833922) [ 2673.953907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2675.575099] Lustre: Failing over lustre-MDT0000 [ 2675.813266] Lustre: server umount lustre-MDT0000 complete [ 2677.729192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2677.742806] LustreError: Skipped 6 previous similar messages [ 2701.011549] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2701.015841] LDISKFS-fs (dm-0): recovery complete [ 2701.027925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.477328] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 2709.252821] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2710.190749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2710.193755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2719.950492] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2722.128266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2730.998797] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 18:46:25 (1786833985) [ 2739.704848] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2741.900791] Lustre: Failing over lustre-MDT0000 [ 2742.241277] Lustre: server umount lustre-MDT0000 complete [ 2763.226329] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2763.228538] LDISKFS-fs (dm-0): recovery complete [ 2763.235449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.427504] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c61e9c [ 2775.222637] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2777.146264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2777.156511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2784.166774] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2785.707229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2794.607322] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:47:29 (1786834049) [ 2801.589432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2803.284035] Lustre: Failing over lustre-MDT0000 [ 2803.469703] Lustre: server umount lustre-MDT0000 complete [ 2823.587727] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2823.592481] LDISKFS-fs (dm-0): recovery complete [ 2823.598576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.962345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2823.967200] Lustre: Skipped 9 previous similar messages [ 2828.169711] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2829.381200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2829.381472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2837.106882] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.715882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2846.658536] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:48:22 (1786834102) [ 2853.211196] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2854.964505] Lustre: Failing over lustre-MDT0000 [ 2855.159724] Lustre: server umount lustre-MDT0000 complete [ 2877.086917] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2877.088929] LDISKFS-fs (dm-0): recovery complete [ 2877.111451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.483471] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2890.883561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2890.888647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2901.232590] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.977407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2912.746638] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:49:27 (1786834167) [ 2919.472221] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2921.484330] Lustre: Failing over lustre-MDT0000 [ 2921.574316] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 2921.874647] Lustre: server umount lustre-MDT0000 complete [ 2942.971486] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2942.976329] LDISKFS-fs (dm-0): recovery complete [ 2942.983903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.226402] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c62fb3 [ 2952.435215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.438328] Lustre: Skipped 9 previous similar messages [ 2956.797370] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2957.911437] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2957.917379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2965.716990] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2967.137725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2975.580723] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:50:30 (1786834230) [ 2981.924663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2983.678913] Lustre: Failing over lustre-MDT0000 [ 2983.973249] Lustre: server umount lustre-MDT0000 complete [ 3003.745187] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3003.746880] LDISKFS-fs (dm-0): recovery complete [ 3003.751527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.873369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3003.882024] Lustre: Skipped 3 previous similar messages [ 3007.466132] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3009.184553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3009.191918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3016.923063] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3018.210594] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3026.642354] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:51:21 (1786834281) [ 3034.764254] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3037.121663] Lustre: Failing over lustre-MDT0000 [ 3037.350664] Lustre: server umount lustre-MDT0000 complete [ 3060.821943] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3060.825578] LDISKFS-fs (dm-0): recovery complete [ 3060.833442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.522967] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3071.078982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3071.082184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3079.559822] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3080.936833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3089.168944] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:52:24 (1786834344) [ 3096.253671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3098.026782] Lustre: Failing over lustre-MDT0000 [ 3098.229018] Lustre: server umount lustre-MDT0000 complete [ 3119.976873] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3119.979783] LDISKFS-fs (dm-0): recovery complete [ 3119.987061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.778157] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5be61df80 x1873628427381760/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3127.802705] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 29 previous similar messages [ 3127.897256] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up) [ 3131.871656] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3133.427901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3133.444254] Lustre: Skipped 43 previous similar messages [ 3133.499316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3133.500110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3141.023184] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3142.455426] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3149.567586] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 18:53:25 (1786834405) [ 3155.504750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3157.264463] Lustre: Failing over lustre-MDT0000 [ 3157.501827] Lustre: server umount lustre-MDT0000 complete [ 3159.010716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.025871] Lustre: Skipped 39 previous similar messages [ 3178.742596] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3178.750970] LDISKFS-fs (dm-0): recovery complete [ 3178.768362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.616841] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd5c6476f [ 3187.297101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3187.304162] Lustre: Skipped 9 previous similar messages [ 3188.611537] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3190.334580] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3190.334795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3197.494457] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3198.893462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3206.796358] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 18:54:22 (1786834462) [ 3207.836964] Lustre: 98898:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting aea50010-778e-4c91-a65b-6c2b6549a881 at adminstrative request [ 3216.692936] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 18:54:32 (1786834472) [ 3221.843849] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3223.394442] Lustre: Failing over lustre-MDT0000 [ 3223.696721] Lustre: server umount lustre-MDT0000 complete [ 3233.927276] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3233.929236] LDISKFS-fs (dm-0): recovery complete [ 3233.939843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.248825] Lustre: lustre-MDT0000: Aborting client recovery [ 3234.251533] LustreError: 100180:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3234.256139] Lustre: 100212:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3234.268129] Lustre: 100212:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client aea50010-778e-4c91-a65b-6c2b6549a881@ [ 3234.293437] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3234.301116] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3234.319481] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3234.376368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 3234.409895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 3237.782312] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3239.402518] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3256.580866] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 18:55:11 (1786834511) [ 3262.473878] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3263.770554] Lustre: Failing over lustre-MDT0000 [ 3263.965616] Lustre: server umount lustre-MDT0000 complete [ 3264.098586] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.119482] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 269 previous similar messages [ 3273.364074] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3273.367640] LDISKFS-fs (dm-0): recovery complete [ 3273.374504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.453344] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.460284] LustreError: Skipped 10 previous similar messages [ 3273.665903] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3273.686218] Lustre: lustre-MDT0000: Aborting client recovery [ 3273.689320] LustreError: 102043:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3273.691328] Lustre: 102076:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3273.698847] Lustre: 102076:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3273.703060] Lustre: 102076:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3273.707851] Lustre: 102076:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3273.711994] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3273.717913] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3273.727827] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3273.766270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3273.769735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3277.167823] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3278.824857] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3290.078947] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3290.083631] Lustre: Skipped 1 previous similar message [ 3295.380176] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 18:55:50 (1786834550) [ 3300.995063] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3302.404110] Lustre: Failing over lustre-MDT0000 [ 3302.560699] Lustre: server umount lustre-MDT0000 complete [ 3304.417908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3304.425293] LustreError: Skipped 8 previous similar messages [ 3311.891421] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3311.893568] LDISKFS-fs (dm-0): recovery complete [ 3311.900224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.210629] Lustre: lustre-MDT0000: Aborting client recovery [ 3312.213131] LustreError: 103909:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3312.218766] Lustre: 103941:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3312.223415] Lustre: 103941:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3312.229115] Lustre: 103941:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3312.236203] Lustre: 103941:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3312.241955] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3312.250813] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3312.261987] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3312.304036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 3312.310041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 3315.459367] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3317.235502] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3331.598543] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 18:56:27 (1786834587) [ 3332.465865] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3332.472136] LustreError: 24767:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb592cfce00 x1873628416361344/t201863462916(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:261/0 lens 512/456 e 0 to 0 dl 1786834601 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3335.739982] Lustre: Failing over lustre-MDT0000 [ 3335.783450] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 3335.929898] Lustre: server umount lustre-MDT0000 complete [ 3342.756749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.018123] Lustre: lustre-MDT0000: Aborting client recovery [ 3343.021328] LustreError: 105232:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3343.028529] Lustre: 105265:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3343.034302] Lustre: 105265:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3343.039381] Lustre: 105265:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3343.048422] Lustre: 105265:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3343.055476] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3343.061101] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3343.070586] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3343.113205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 3343.119140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3347.015204] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3348.452293] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3363.930904] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3365.154693] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 18:57:00 (1786834620) [ 3370.575945] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3372.635788] Lustre: Failing over lustre-MDT0000 [ 3372.789967] Lustre: server umount lustre-MDT0000 complete [ 3382.156199] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3382.159704] LDISKFS-fs (dm-0): recovery complete [ 3382.171081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.473138] Lustre: lustre-MDT0000: Aborting client recovery [ 3382.475447] LustreError: 107199:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3382.479608] Lustre: 107231:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3382.483103] Lustre: 107231:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3382.487205] Lustre: 107231:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3382.492852] Lustre: 107231:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3382.496770] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3382.501082] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3382.519795] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3382.552224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3382.555617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 3385.838406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3387.901937] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3402.731100] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:57:38 (1786834658) [ 3430.087621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3431.435547] Lustre: Failing over lustre-MDT0000 [ 3431.752947] Lustre: server umount lustre-MDT0000 complete [ 3449.312669] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786834691/real 1786834691] req@ffff8bb486754380 x1873628427654656/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786834707 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3449.340307] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 3450.981453] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3450.984060] LDISKFS-fs (dm-0): recovery complete [ 3450.992643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.556946] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5c1bebb80 x1873628427663232/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3459.803391] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3459.808161] Lustre: Skipped 21 previous similar messages [ 3462.624402] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3465.269205] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3465.275697] Lustre: Skipped 10 previous similar messages [ 3465.296981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3465.306151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3469.077454] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3470.238859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3486.493925] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:59:02 (1786834742) [ 3505.019292] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3512.364553] Lustre: Failing over lustre-MDT0000 [ 3512.643974] Lustre: server umount lustre-MDT0000 complete [ 3532.075732] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3532.078460] LDISKFS-fs (dm-0): recovery complete [ 3532.085817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.557475] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3539.690406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3539.692199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3544.156235] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3545.304969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3562.967529] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 19:00:18 (1786834818) [ 3564.747442] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3565.507524] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3565.519159] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3570.070407] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 19:00:25 (1786834825) [ 3590.204461] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3599.081578] Lustre: Failing over lustre-OST0000 [ 3599.163373] Lustre: server umount lustre-OST0000 complete [ 3618.584145] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3618.586083] LDISKFS-fs (dm-2): recovery complete [ 3618.590883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3618.728654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.736870] Lustre: Skipped 11 previous similar messages [ 3623.349375] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3676.369854] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 19:02:12 (1786834932) [ 3681.525218] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3683.268081] Lustre: Failing over lustre-MDT0000 [ 3683.296959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.304690] Lustre: Skipped 1 previous similar message [ 3683.422946] Lustre: server umount lustre-MDT0000 complete [ 3702.917978] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3702.920521] LDISKFS-fs (dm-0): recovery complete [ 3702.938262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.969553] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5a9145f80 x1873628428141312/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3715.319159] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3717.663377] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3717.664944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3721.503164] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3722.497385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3733.984204] LustreError: 116643:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3733.985640] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3734.001572] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3734.007974] Lustre: Skipped 43 previous similar messages [ 3735.009152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3738.649711] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 19:03:14 (1786834994) [ 3742.464377] LustreError: 8677:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3747.808402] LustreError: 8677:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3747.814377] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3749.462934] LustreError: 24767:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3754.976990] LustreError: 24767:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3754.982937] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3756.309280] LustreError: 6490:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3761.637055] LustreError: 6490:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3761.644199] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3763.006308] LustreError: 24767:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3768.289696] LustreError: 24767:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3769.610309] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3774.944138] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3774.949125] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3774.963151] Lustre: Skipped 1 previous similar message [ 3783.075529] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3783.085045] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3788.256116] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3788.262056] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3794.912165] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3794.918884] Lustre: Skipped 2 previous similar messages [ 3802.741859] LustreError: 9505:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3802.748055] LustreError: 9505:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3808.225888] LustreError: 9505:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3808.231868] LustreError: 9505:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3813.705344] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 19:04:29 (1786835069) [ 3814.928791] LustreError: 6491:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3825.244133] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3830.363250] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3835.486153] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3840.602293] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3840.607361] Lustre: Skipped 1 previous similar message [ 3845.725595] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3855.000147] LustreError: 6491:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3855.005733] Lustre: 6491:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8bb5aecf9880 x1873628419012224/t0(0) o38->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:0/0 lens 520/416 e 0 to 0 dl 1786835092 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3855.963859] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3855.969654] Lustre: Skipped 3 previous similar messages [ 3855.973476] LustreError: 8677:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3880.538541] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3880.542599] Lustre: Skipped 1 previous similar message [ 3896.032117] LustreError: 8677:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3896.039251] Lustre: 8677:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8bb4899d6d80 x1873628419016576/t0(0) o38->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:0/0 lens 520/416 e 0 to 0 dl 1786835133 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3901.019979] LustreError: 6491:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3925.595378] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3925.602385] Lustre: Skipped 3 previous similar messages [ 3941.080104] LustreError: 6491:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3941.088824] Lustre: 6491:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff8bb5b77bdf80 x1873628419020288/t0(0) o38->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:0/0 lens 520/416 e 0 to 0 dl 1786835178 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3945.656794] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 3945.666684] Lustre: Skipped 1 previous similar message [ 3945.671725] LustreError: 8677:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3971.169120] Lustre: lustre-MDT0000: Export ffff8bb5a9104800 already connecting from 192.168.203.35@tcp [ 3971.178881] Lustre: Skipped 3 previous similar messages [ 3985.744270] LustreError: 8677:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3985.751924] Lustre: 8677:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8bb4899d6680 x1873628419023616/t0(0) o38->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:0/0 lens 520/416 e 0 to 0 dl 1786835223 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3986.522631] LustreError: 6490:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4026.584229] LustreError: 6490:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4026.591744] Lustre: 6490:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8bb48a4d7100 x1873628419026432/t0(0) o38->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:0/0 lens 520/416 e 0 to 0 dl 1786835264 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4027.483695] LustreError: 6492:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4031.648341] LustreError: 6492:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout interrupted [ 4034.074668] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 19:08:09 (1786835289) [ 4037.954760] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4040.300879] Lustre: Failing over lustre-MDT0000 [ 4040.431983] Lustre: server umount lustre-MDT0000 complete [ 4041.187966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.191637] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.198879] Lustre: Skipped 38 previous similar messages [ 4041.209522] LustreError: 7971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 136 previous similar messages [ 4046.754615] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4046.756580] LDISKFS-fs (dm-0): recovery complete [ 4046.762644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.855097] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4046.863292] LustreError: Skipped 6 previous similar messages [ 4046.953763] Lustre: *** cfs_fail_loc=712, val=0*** [ 4046.957405] LustreError: 43878:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8bb5b7757100 x1873628428345344/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4047.030518] Lustre: lustre-MDT0000: Aborting client recovery [ 4047.033299] LustreError: 121483:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4047.036904] Lustre: 121515:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4047.041471] Lustre: 121515:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 4047.044532] Lustre: 121515:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4047.048262] Lustre: 121515:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4047.051149] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4047.055464] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4047.064141] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4047.096033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4047.098148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4048.961957] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4052.454542] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4061.161949] Lustre: Failing over lustre-MDT0000 [ 4061.322840] Lustre: server umount lustre-MDT0000 complete [ 4062.691928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4062.698590] LustreError: Skipped 7 previous similar messages [ 4076.167165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.393449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4076.400030] Lustre: Skipped 6 previous similar messages [ 4078.609789] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4078.680943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4078.687525] Lustre: Skipped 4 previous similar messages [ 4081.652149] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4081.657884] Lustre: Skipped 3 previous similar messages [ 4081.686215] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4081.686241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4084.723239] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4085.604957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4091.105892] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 19:09:06 (1786835346) [ 4091.167239] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 4091.171509] Lustre: Skipped 2 previous similar messages [ 4096.339830] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 19:09:12 (1786835352) [ 4096.851233] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4096.853651] LustreError: 6499:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5b43ac700 x1873628419084416/t0(0) o700->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:270/0 lens 264/248 e 0 to 0 dl 1786835365 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4114.744258] Lustre: Failing over lustre-MDT0000 [ 4114.871371] Lustre: server umount lustre-MDT0000 complete [ 4129.082256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.443584] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4134.409706] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4134.413277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4137.499514] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4138.379438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4143.815560] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 19:09:59 (1786835399) [ 4145.080089] Lustre: Failing over lustre-OST0000 [ 4145.163520] Lustre: server umount lustre-OST0000 complete [ 4159.758612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4163.001601] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4168.633577] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4169.524630] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4236.835042] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 19:11:32 (1786835492) [ 4240.781843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4242.166674] Lustre: Failing over lustre-MDT0000 [ 4242.345660] Lustre: server umount lustre-MDT0000 complete [ 4258.576316] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4258.578270] LDISKFS-fs (dm-0): recovery complete [ 4258.592963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.781067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.784049] Lustre: Skipped 5 previous similar messages [ 4260.946338] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4264.097922] Lustre: *** cfs_fail_loc=216, val=0*** [ 4264.099139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4264.099831] LustreError: 127641:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4265.120702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4327.807766] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 19:13:03 (1786835583) [ 4337.346758] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 19:13:13 (1786835593) [ 4338.883180] Lustre: Failing over lustre-MDT0000 [ 4339.044344] Lustre: server umount lustre-MDT0000 complete [ 4353.256331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4355.369546] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4358.626220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4358.630347] Lustre: Skipped 20 previous similar messages [ 4358.636992] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4358.639335] LustreError: 129219:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5bf854000 x1873628419204096/t0(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:573/0 lens 328/344 e 0 to 0 dl 1786835668 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4416.092099] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 4416.102598] Lustre: 129219:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4416.139646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4416.139759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4418.738302] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4419.438839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4424.306799] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 19:14:40 (1786835680) [ 4425.825128] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4429.269103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4430.059711] Lustre: Failing over lustre-MDT0000 [ 4430.215914] Lustre: server umount lustre-MDT0000 complete [ 4445.136708] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4445.138629] LDISKFS-fs (dm-0): recovery complete [ 4445.143878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.995982] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4450.810166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 4450.810197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4453.095591] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4453.736286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4457.981935] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 19:15:13 (1786835713) [ 4458.553904] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4463.477950] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4464.419231] Lustre: Failing over lustre-MDT0000 [ 4466.552933] Lustre: server umount lustre-MDT0000 complete [ 4482.402586] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4482.404946] LDISKFS-fs (dm-0): recovery complete [ 4482.412274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.665539] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4487.682153] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4487.682345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 4490.020855] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4490.706642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4494.766621] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 19:15:50 (1786835750) [ 4495.230959] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4499.769150] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4500.567216] Lustre: Failing over lustre-MDT0000 [ 4500.654784] Lustre: server umount lustre-MDT0000 complete [ 4516.038451] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4516.040650] LDISKFS-fs (dm-0): recovery complete [ 4516.044780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.997488] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4521.466249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 4521.466615] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 4526.672582] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 19:16:22 (1786835782) [ 4528.201730] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4528.203786] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4528.208210] LustreError: 6493:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5b0ecd180 x1873628419254784/t257698037777(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:702/0 lens 392/456 e 0 to 0 dl 1786835797 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4529.536127] Lustre: Failing over lustre-MDT0000 [ 4529.712053] Lustre: server umount lustre-MDT0000 complete [ 4543.499386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.469829] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4549.109608] Lustre: 43596:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb48a11c000 x1873628419254784/t257698037777(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:723/0 lens 392/456 e 0 to 0 dl 1786835818 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4549.115268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 4549.115510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 4551.173977] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4551.870699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4555.691208] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 19:16:51 (1786835811) [ 4556.159329] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4556.163697] LustreError: 6490:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5aed8df80 x1873628419267584/t261993005072(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:19/0 lens 504/448 e 0 to 0 dl 1786835869 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4560.579146] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4561.360904] Lustre: Failing over lustre-MDT0000 [ 4561.458747] Lustre: server umount lustre-MDT0000 complete [ 4576.464976] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4576.466545] LDISKFS-fs (dm-0): recovery complete [ 4576.471534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.416175] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4581.895908] Lustre: 6492:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb588c54e00 x1873628419267584/t261993005072(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:44/0 lens 504/2880 e 0 to 0 dl 1786835894 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4581.898038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3265) [ 4581.898765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 4584.194353] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4584.808245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4588.668651] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 19:17:24 (1786835844) [ 4589.109812] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4589.112408] LustreError: 6492:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb48a11ce00 x1873628419281408/t266287972368(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:52/0 lens 504/448 e 0 to 0 dl 1786835902 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4590.477248] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4593.370741] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4594.164891] Lustre: Failing over lustre-MDT0000 [ 4594.278221] Lustre: server umount lustre-MDT0000 complete [ 4608.753804] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4608.755375] LDISKFS-fs (dm-0): recovery complete [ 4608.759095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.389574] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4614.132037] Lustre: 8677:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5b0eef100 x1873628419281408/t266287972368(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:77/0 lens 504/2880 e 0 to 0 dl 1786835927 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4614.137317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3297) [ 4614.137339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4614.149546] Lustre: 8677:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4619.049484] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 19:17:55 (1786835875) [ 4619.463475] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4619.465096] Lustre: Skipped 1 previous similar message [ 4619.466424] LustreError: 9505:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5b88b1500 x1873628419294080/t270582939664(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:82/0 lens 504/448 e 0 to 0 dl 1786835932 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4619.473921] LustreError: 9505:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 4620.785859] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4620.788352] Lustre: Skipped 1 previous similar message [ 4623.877612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4624.601492] Lustre: Failing over lustre-MDT0000 [ 4624.721571] Lustre: server umount lustre-MDT0000 complete [ 4639.703913] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4639.706019] LDISKFS-fs (dm-0): recovery complete [ 4639.710324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4641.614922] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4645.362273] Lustre: 6490:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb48a11e300 x1873628419294080/t270582939664(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:108/0 lens 504/2880 e 0 to 0 dl 1786835958 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4645.368939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4645.369592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3329) [ 4649.975230] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 19:18:25 (1786835905) [ 4650.412876] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4651.756652] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4651.758288] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4651.759813] LustreError: 6493:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5b1f46050 x1873628419306880/t274877906960(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:70/0 lens 392/456 e 0 to 0 dl 1786835920 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4655.666188] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4656.416268] Lustre: Failing over lustre-MDT0000 [ 4656.507423] Lustre: server umount lustre-MDT0000 complete [ 4656.743034] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.752495] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 179 previous similar messages [ 4660.704944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4660.709709] Lustre: Skipped 52 previous similar messages [ 4671.353682] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4671.355598] LDISKFS-fs (dm-0): recovery complete [ 4671.362699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.424572] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4671.427722] LustreError: Skipped 11 previous similar messages [ 4673.140054] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4676.605770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 4676.605891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3361) [ 4676.606631] Lustre: 6494:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5b7756a00 x1873628419306880/t274877906960(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:95/0 lens 392/456 e 0 to 0 dl 1786835945 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4681.639861] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 19:18:57 (1786835937) [ 4682.058693] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4741.738117] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnecting [ 4741.741818] Lustre: Skipped 4 previous similar messages [ 4741.745658] Lustre: 6492:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5b0eece00 x1873628419316864/t279172874255(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:204/0 lens 664/3488 e 0 to 0 dl 1786836054 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4744.570267] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 19:20:00 (1786836000) [ 4747.774570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4748.530640] Lustre: Failing over lustre-MDT0000 [ 4748.636541] Lustre: server umount lustre-MDT0000 complete [ 4753.377201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4753.379935] LustreError: Skipped 11 previous similar messages [ 4763.513535] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4763.515410] LDISKFS-fs (dm-0): recovery complete [ 4763.520050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.617227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4763.623063] Lustre: Skipped 3 previous similar messages [ 4763.691452] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4763.694440] Lustre: Skipped 12 previous similar messages [ 4765.116341] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4767.321497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4767.325679] Lustre: Skipped 12 previous similar messages [ 4768.754420] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4768.758201] Lustre: Skipped 12 previous similar messages [ 4768.775408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 4768.775423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3393) [ 4770.841797] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4771.439481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4785.478973] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 19:20:41 (1786836041) [ 4788.652128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4789.396319] Lustre: Failing over lustre-MDT0000 [ 4789.536439] Lustre: server umount lustre-MDT0000 complete [ 4804.315659] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4804.317305] LDISKFS-fs (dm-0): recovery complete [ 4804.322377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.184287] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4809.733403] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 4809.735519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3425) [ 4811.809642] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.466631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4815.020377] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4820.999266] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 19:21:16 (1786836076) [ 4837.903841] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4838.729663] Lustre: Failing over lustre-MDT0000 [ 4838.984919] Lustre: server umount lustre-MDT0000 complete [ 4853.664171] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4853.665813] LDISKFS-fs (dm-0): recovery complete [ 4853.673154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.515313] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4858.996660] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4858.996833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4861.204429] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4861.791769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4881.332100] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 19:22:17 (1786836137) [ 4884.963485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4885.732892] Lustre: Failing over lustre-MDT0000 [ 4885.862182] Lustre: server umount lustre-MDT0000 complete [ 4900.132638] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4900.134351] LDISKFS-fs (dm-0): recovery complete [ 4900.138790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.267077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.269626] Lustre: Skipped 12 previous similar messages [ 4901.632722] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4905.468573] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 4905.468581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 4907.202754] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4907.690809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4911.127067] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 4911.658601] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4913.899192] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 19:22:50 (1786836170) [ 4919.787741] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4981.259958] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4981.262063] Lustre: Skipped 1 previous similar message [ 4981.263993] LustreError: 24767:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5a9145f80 x1873628422106880/t296352743435(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:444/0 lens 66040/440 e 0 to 0 dl 1786836294 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4981.273156] LustreError: 24767:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 5040.730594] Lustre: 6492:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5c142c850 x1873628422106880/t296352743435(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:503/0 lens 66040/440 e 0 to 0 dl 1786836353 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5044.145378] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5044.740273] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 19:25:00 (1786836300) [ 5049.667491] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5050.519070] Lustre: Failing over lustre-MDT0000 [ 5050.715582] Lustre: server umount lustre-MDT0000 complete [ 5065.143860] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5065.145244] LDISKFS-fs (dm-0): recovery complete [ 5065.149095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.789977] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5070.306835] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5070.309776] Lustre: Skipped 51 previous similar messages [ 5070.554808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5070.554807] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5072.577327] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5073.204208] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5077.390196] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 19:25:33 (1786836333) [ 5084.135857] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5087.985857] Lustre: Failing over lustre-OST0000 [ 5088.025201] Lustre: server umount lustre-OST0000 complete [ 5102.425096] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5102.426909] LDISKFS-fs (dm-2): recovery complete [ 5102.430879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5104.619904] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5116.301276] Lustre: Failing over lustre-OST0000 [ 5116.349157] Lustre: server umount lustre-OST0000 complete [ 5129.486155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.682965] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5135.844143] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5136.485782] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5170.726078] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 19:27:06 (1786836426) [ 5171.731282] Lustre: Failing over lustre-MDT0000 [ 5171.963400] Lustre: server umount lustre-MDT0000 complete [ 5185.084304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.799298] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5190.646093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5190.646104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5198.466198] Lustre: Failing over lustre-MDT0000 [ 5198.628755] Lustre: server umount lustre-MDT0000 complete [ 5211.613119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.191207] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5217.273268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5217.273299] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5219.193422] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5219.752857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5223.510862] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 19:27:59 (1786836479) [ 5234.969902] Lustre: Failing over lustre-OST0000 [ 5235.011974] Lustre: server umount lustre-OST0000 complete [ 5247.764659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5249.881131] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5253.526757] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5254.046980] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5257.851698] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 19:28:34 (1786836514) [ 5258.524509] Lustre: Failing over lustre-MDT0000 [ 5258.766040] Lustre: server umount lustre-MDT0000 complete [ 5260.898968] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.906793] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 136 previous similar messages [ 5261.889600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.922121] Lustre: *** cfs_fail_loc=605, val=0*** [ 5261.923437] LustreError: 164621:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc09f34b0 failed: rc = -95 [ 5261.926117] LustreError: 164621:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5261.927955] LustreError: 164621:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 5261.930032] LustreError: 164621:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5261.932261] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5261.934642] LustreError: 164621:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5261.939925] Lustre: server umount lustre-MDT0000 complete [ 5261.941023] LustreError: 164621:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5263.328992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.332899] Lustre: Skipped 34 previous similar messages [ 5264.669645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5266.238947] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5269.190475] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 19:28:45 (1786836525) [ 5270.004469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5270.004469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5272.534108] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5273.620307] Lustre: Failing over lustre-MDT0000 [ 5273.861578] Lustre: server umount lustre-MDT0000 complete [ 5287.930487] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5287.933441] LDISKFS-fs (dm-0): recovery complete [ 5287.939055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.982305] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.985995] LustreError: Skipped 8 previous similar messages [ 5289.536667] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5291.613420] Lustre: *** cfs_fail_loc=707, val=0*** [ 5352.047722] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5352.059984] Lustre: 166789:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5352.066243] Lustre: 166789:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 5352.197202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 5352.197339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 5354.322976] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5354.924261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5359.078065] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 19:30:15 (1786836615) [ 5382.799718] LustreError: 6490:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb5be623100 x1873628422987008/t0(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:46/0 lens 664/0 e 0 to 0 dl 1786836651 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5382.806460] LustreError: 6490:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5393.832062] LustreError: 6490:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5393.837403] LustreError: 43596:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb5b0e63800 x1873628422988288/t0(0) o35->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:57/0 lens 392/0 e 0 to 0 dl 1786836662 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5400.034037] LustreError: 43882:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb48f6db100 x1873628430065024/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:63/0 lens 544/0 e 0 to 0 dl 1786836668 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 5400.043329] LustreError: 43882:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 5405.708582] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 19:31:01 (1786836661) [ 5429.157399] LustreError: 42719:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5440.184088] LustreError: 42719:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5443.359655] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 19:31:39 (1786836699) [ 5466.354288] LustreError: 6492:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb5b0eb8700 x1873628423011712/t0(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:130/0 lens 576/0 e 0 to 0 dl 1786836735 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5466.361428] LustreError: 6492:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5471.464098] LustreError: 6492:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5471.468420] LustreError: 6490:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb5be621880 x1873628423012480/t0(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:135/0 lens 664/0 e 0 to 0 dl 1786836740 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5482.360076] LustreError: 9505:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5482.365285] LustreError: 24767:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb592cfd880 x1873628423033984/t0(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:185/0 lens 664/0 e 0 to 0 dl 1786836790 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5482.372139] LustreError: 24767:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5494.316409] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 19:32:30 (1786836750) [ 5579.396020] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 19:33:55 (1786836835) [ 5602.590218] LustreError: 8677:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb5b0eedf80 x1873628423087232/t0(0) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:266/0 lens 576/0 e 0 to 0 dl 1786836871 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5602.597992] LustreError: 8677:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 5602.601205] LustreError: 8677:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5602.606623] LustreError: 8677:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5603.032107] LustreError: 8677:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5618.890937] LustreError: 6491:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5618.894694] LustreError: 6491:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5619.312108] LustreError: 6491:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5619.315623] LustreError: 6491:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5634.754644] LustreError: 6491:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8bb48764b100 x1873628423106176/t0(0) o36->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:298/0 lens 504/0 e 0 to 0 dl 1786836903 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5634.763723] LustreError: 6491:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 81 previous similar messages [ 5646.778254] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 19:35:03 (1786836903) [ 5671.185402] Lustre: DEBUG MARKER: phase 2 [ 5674.274519] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 19:35:30 (1786836930) [ 5744.915307] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 19:36:40 (1786837000) [ 5745.577601] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5746.257883] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:36:42 (1786837002) [ 5748.360558] Lustre: DEBUG MARKER: Started rundbench load pid=135075 ... [ 5752.425674] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5754.094474] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5754.978430] Lustre: Failing over lustre-MDT0000 [ 5754.997022] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 5755.235829] Lustre: server umount lustre-MDT0000 complete [ 5756.896531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5756.899165] LustreError: Skipped 8 previous similar messages [ 5769.794256] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5769.796053] LDISKFS-fs (dm-0): recovery complete [ 5769.802761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.947635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5769.950504] Lustre: Skipped 8 previous similar messages [ 5769.964239] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5769.967984] Lustre: Skipped 11 previous similar messages [ 5771.423549] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5772.888900] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5772.892192] Lustre: Skipped 11 previous similar messages [ 5775.331131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5775.334446] Lustre: Skipped 25 previous similar messages [ 5775.554223] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5775.557957] Lustre: Skipped 11 previous similar messages [ 5775.576841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5091 to 0x280000401:5121) [ 5775.576967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5054 to 0x2c0000401:5089) [ 5777.636360] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5778.209892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5783.524213] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5785.135032] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5785.895804] Lustre: Failing over lustre-MDT0001 [ 5785.991102] Lustre: server umount lustre-MDT0001 complete [ 5800.600961] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5800.602958] LDISKFS-fs (dm-1): recovery complete [ 5800.609995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.203763] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5806.071915] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) [ 5806.071921] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) [ 5808.176718] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.795294] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5814.339427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5815.981277] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5816.783863] Lustre: Failing over lustre-MDT0000 [ 5816.951350] Lustre: server umount lustre-MDT0000 complete [ 5831.601817] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5831.603271] LDISKFS-fs (dm-0): recovery complete [ 5831.607835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.295546] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5837.344924] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5384 to 0x2c0000401:5409) [ 5837.345260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5416 to 0x280000401:5441) [ 5839.487726] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5840.130468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5845.485968] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5847.140904] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 5847.896219] Lustre: Failing over lustre-MDT0001 [ 5848.061428] Lustre: server umount lustre-MDT0001 complete [ 5862.368683] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.374369] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 70 previous similar messages [ 5862.606573] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5862.608436] LDISKFS-fs (dm-1): recovery complete [ 5862.616107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.253589] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5868.022402] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) [ 5868.022460] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) [ 5870.050095] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5870.646922] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5894.502091] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 19:39:10 (1786837150) [ 6017.576489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6028.400141] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6029.131563] Lustre: Failing over lustre-MDT0000 [ 6029.396617] Lustre: server umount lustre-MDT0000 complete [ 6031.841163] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6031.845043] Lustre: Skipped 24 previous similar messages [ 6043.652018] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6043.653280] LDISKFS-fs (dm-0): recovery complete [ 6043.656324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6043.701343] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6043.706615] LustreError: Skipped 2 previous similar messages [ 6045.170416] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6051.383399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6841 to 0x280000401:6881) [ 6051.383462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6809 to 0x2c0000401:6849) [ 6053.381140] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6053.973899] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6178.288783] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6189.075854] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 6189.749941] Lustre: Failing over lustre-MDT0001 [ 6189.989573] Lustre: server umount lustre-MDT0001 complete [ 6204.206057] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6204.208407] LDISKFS-fs (dm-1): recovery complete [ 6204.213899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.640191] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6211.886303] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2516 to 0x280000400:2561) [ 6211.886477] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2515 to 0x2c0000400:2561) [ 6213.916420] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6214.530207] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6232.626619] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 19:44:48 (1786837488) [ 6355.679123] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6366.416401] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 6367.029774] Lustre: Failing over lustre-MDT0001 [ 6368.225074] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6368.227187] Lustre: Skipped 5 previous similar messages [ 6372.942921] Lustre: server umount lustre-MDT0001 complete [ 6386.789209] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6386.790282] LDISKFS-fs (dm-1): recovery complete [ 6386.794490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.883973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.886456] Lustre: Skipped 5 previous similar messages [ 6386.893089] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6386.894810] Lustre: Skipped 5 previous similar messages [ 6388.105758] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6388.827055] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6388.829560] Lustre: Skipped 5 previous similar messages [ 6392.289717] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6392.291992] Lustre: Skipped 20 previous similar messages [ 6393.268643] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6393.272935] Lustre: Skipped 5 previous similar messages [ 6393.291571] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2955 to 0x2c0000400:2977) [ 6393.292123] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2955 to 0x280000400:2977) [ 6395.144514] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6395.706621] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6519.696813] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6530.434190] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 6531.052060] Lustre: Failing over lustre-MDT0000 [ 6531.365897] Lustre: server umount lustre-MDT0000 complete [ 6532.696419] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.701204] LustreError: 9505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 6535.648804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6535.650782] LustreError: Skipped 5 previous similar messages [ 6545.049438] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6545.050872] LDISKFS-fs (dm-0): recovery complete [ 6545.053477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.323088] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6551.350370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8826 to 0x280000401:8865) [ 6551.350689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8793 to 0x2c0000401:8833) [ 6553.180688] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6553.702788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6557.019163] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 19:50:13 (1786837813) [ 6680.055443] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6690.716897] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 6691.282904] Lustre: Failing over lustre-MDT0001 [ 6691.303076] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.307854] Lustre: Skipped 10 previous similar messages [ 6691.427796] Lustre: server umount lustre-MDT0001 complete [ 6705.378624] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6705.380675] LDISKFS-fs (dm-1): recovery complete [ 6705.385173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.882775] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6710.774227] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3413 to 0x280000400:3457) [ 6710.774234] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3414 to 0x2c0000400:3457) [ 6712.732709] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6713.337106] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6837.467771] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6848.241495] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 6848.871252] Lustre: Failing over lustre-MDT0001 [ 6849.017568] Lustre: server umount lustre-MDT0001 complete [ 6862.834476] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6862.835503] LDISKFS-fs (dm-1): recovery complete [ 6862.838739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.125411] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6867.960517] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3413 to 0x280000400:3489) [ 6867.960574] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3414 to 0x2c0000400:3489) [ 6869.923599] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6870.565618] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6874.403689] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 19:55:30 (1786838130) [ 6880.145457] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6881.620905] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6882.224964] Lustre: Failing over lustre-OST0000 [ 6882.266730] Lustre: server umount lustre-OST0000 complete [ 6895.956524] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6895.957720] LDISKFS-fs (dm-2): recovery complete [ 6895.960815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.743527] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6901.475429] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.974977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6910.622714] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6912.084788] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6912.666841] Lustre: Failing over lustre-OST0000 [ 6912.700065] Lustre: server umount lustre-OST0000 complete [ 6926.314060] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6926.315908] LDISKFS-fs (dm-2): recovery complete [ 6926.318824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.136300] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6931.866358] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6932.419700] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6937.793212] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 19:56:34 (1786838194) [ 7060.583419] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7063.062610] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7073.759540] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7074.368171] Lustre: Failing over lustre-MDT0001 [ 7074.491801] Lustre: server umount lustre-MDT0001 complete [ 7075.624111] Lustre: Failing over lustre-MDT0000 [ 7075.678893] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 7075.681363] Lustre: Skipped 4 previous similar messages [ 7081.079624] Lustre: server umount lustre-MDT0000 complete [ 7095.518554] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7095.522455] LDISKFS-fs (dm-1): recovery complete [ 7095.529040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.554176] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7095.556194] LDISKFS-fs (dm-0): recovery complete [ 7095.559860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7095.601369] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7095.604246] LustreError: Skipped 1 previous similar message [ 7095.687618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7095.689749] Lustre: Skipped 5 previous similar messages [ 7095.697551] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7095.699551] Lustre: Skipped 5 previous similar messages [ 7097.225779] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7097.368265] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7099.873170] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7099.874418] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7099.875414] Lustre: Skipped 6 previous similar messages [ 7099.877509] Lustre: Skipped 16 previous similar messages [ 7100.838397] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7100.840936] Lustre: Skipped 5 previous similar messages [ 7100.855139] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3414 to 0x2c0000400:3521) [ 7100.855139] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3413 to 0x280000400:3521) [ 7109.649424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12951 to 0x280000401:12993) [ 7109.653054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12919 to 0x2c0000401:12961) [ 7111.612658] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7112.128820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7112.668906] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7237.989113] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7241.872423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7252.976891] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 7254.003250] Lustre: Failing over lustre-MDT0001 [ 7254.015088] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7254.020328] LustreError: Skipped 7 previous similar messages [ 7254.155868] Lustre: server umount lustre-MDT0001 complete [ 7254.498784] LustreError: 198433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.509038] LustreError: 198433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 59 previous similar messages [ 7255.853150] Lustre: Failing over lustre-MDT0000 [ 7262.340567] Lustre: server umount lustre-MDT0000 complete [ 7279.012831] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7279.016431] LDISKFS-fs (dm-1): recovery complete [ 7279.030422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.100977] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7279.103394] LDISKFS-fs (dm-0): recovery complete [ 7279.109538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.197843] LustreError: 201828:0:(llog.c:1655:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7279.198355] LustreError: 201825:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.135@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8bb48aabae00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7279.203844] Lustre: 201828:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7281.401685] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7281.498220] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7285.514953] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3413 to 0x280000400:3553) [ 7285.515227] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3414 to 0x2c0000400:3553) [ 7291.199456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12919 to 0x2c0000401:12993) [ 7291.199874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12951 to 0x280000401:13025) [ 7294.630410] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7295.461928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7296.294308] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7300.876436] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 20:02:36 (1786838556) [ 7304.632971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7305.829085] Lustre: Failing over lustre-MDT0000 [ 7305.992394] Lustre: server umount lustre-MDT0000 complete [ 7306.720725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.727979] Lustre: Skipped 19 previous similar messages [ 7321.602928] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7321.605231] LDISKFS-fs (dm-0): recovery complete [ 7321.609914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7323.540199] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7325.276060] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7340.645467] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7340.663748] Lustre: 201836:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb491501880 x1873628499730560/t335007490945(335007490945) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:494/0 lens 520/3488 e 0 to 0 dl 1786838609 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7340.724990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13027 to 0x280000401:13057) [ 7340.725172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12919 to 0x2c0000401:13025) [ 7343.261490] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7343.975195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7348.020533] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 20:03:24 (1786838604) [ 7351.104842] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7352.283416] Lustre: Failing over lustre-MDT0000 [ 7352.414199] Lustre: server umount lustre-MDT0000 complete [ 7368.060489] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7368.061996] LDISKFS-fs (dm-0): recovery complete [ 7368.068731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7370.082054] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7371.364186] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7371.366924] LustreError: 203750:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb489c54380 x1873628499730560/t335007490945(335007490945) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:525/0 lens 520/664 e 0 to 0 dl 1786838640 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7386.731160] Lustre: lustre-MDT0000: Client aea50010-778e-4c91-a65b-6c2b6549a881 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7386.739638] Lustre: 203750:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5b8b49880 x1873628499730560/t335007490945(335007490945) o101->aea50010-778e-4c91-a65b-6c2b6549a881@192.168.203.35@tcp:540/0 lens 520/3488 e 0 to 0 dl 1786838655 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7386.786938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12919 to 0x2c0000401:13057) [ 7386.787028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13089) [ 7389.729369] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7390.483166] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7394.485363] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 20:04:10 (1786838650) [ 7396.927813] Lustre: Failing over lustre-OST0000 [ 7396.981821] Lustre: server umount lustre-OST0000 complete [ 7398.542165] Lustre: Failing over lustre-MDT0000 [ 7398.636200] Lustre: server umount lustre-MDT0000 complete [ 7412.326450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.324392] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7415.353163] Lustre: lustre-MDT0000: Denying connection for new client 778185d9-ba67-4d54-8e96-e0aee8a989d9 (at 192.168.203.35@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 7417.863981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12919 to 0x2c0000401:13089) [ 7422.893398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7425.236866] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7428.076329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13121) [ 7430.337269] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 20:04:46 (1786838686) [ 7430.817781] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7430.819656] LustreError: 202685:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5afd22300 x1873628476267648/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:584/0 lens 1312/4320 e 0 to 0 dl 1786838699 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7434.032886] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7434.935612] Lustre: Failing over lustre-MDT0000 [ 7435.186808] Lustre: server umount lustre-MDT0000 complete [ 7447.008666] Lustre: 201862:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838688/real 1786838688] req@ffff8bb5afd21180 x1873628476267648/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1786838704 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7447.022567] Lustre: 201862:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7451.281706] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7451.283727] LDISKFS-fs (dm-0): recovery complete [ 7451.287869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.831152] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7456.792293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13153) [ 7456.792304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13121) [ 7459.264799] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7459.983937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7464.628117] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 20:05:20 (1786838720) [ 7465.118486] LustreError: 201840:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5afd22300 x1873628476296960/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:619/0 lens 2520/4320 e 0 to 0 dl 1786838734 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7469.022574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7473.069985] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7474.026293] Lustre: Failing over lustre-MDT0001 [ 7479.893677] Lustre: server umount lustre-MDT0001 complete [ 7495.572803] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7495.575385] LDISKFS-fs (dm-1): recovery complete [ 7495.585937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.738560] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7500.792569] Lustre: 201873:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb48aa04e00 x1873628510508800/t38654707045(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:680/0 lens 560/2880 e 0 to 0 dl 1786838795 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7500.802139] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3564 to 0x2c0000400:3585) [ 7500.802241] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3564 to 0x280000400:3585) [ 7503.561610] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7504.508996] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7509.802287] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 20:06:05 (1786838765) [ 7510.310219] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7510.316353] Lustre: Skipped 1 previous similar message [ 7510.321373] LustreError: 201840:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5beefea00 x1873628476330496/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:664/0 lens 2520/4320 e 0 to 0 dl 1786838779 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7513.807233] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7517.446851] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7518.282080] Lustre: Failing over lustre-MDT0000 [ 7518.534773] Lustre: server umount lustre-MDT0000 complete [ 7526.880107] Lustre: 212702:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786838768/real 1786838768] req@ffff8bb5beeff480 x1873628476330496/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1786838784 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7533.797976] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7533.801330] LDISKFS-fs (dm-0): recovery complete [ 7533.805936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.488303] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7539.234059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13153) [ 7539.234059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13185) [ 7541.329526] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7541.990197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7544.091074] Lustre: Failing over lustre-MDT0001 [ 7544.230766] Lustre: server umount lustre-MDT0001 complete [ 7558.997942] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7558.999918] LDISKFS-fs (dm-1): recovery complete [ 7559.008193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.634987] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7564.297693] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3596 to 0x2c0000400:3617) [ 7564.297718] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3596 to 0x280000400:3617) [ 7566.732973] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7567.345863] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7571.496230] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 20:07:07 (1786838827) [ 7577.866652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7581.049778] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7581.840383] Lustre: Failing over lustre-MDT0000 [ 7582.079291] Lustre: server umount lustre-MDT0000 complete [ 7583.544039] LustreError: 151696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838841 with bad export cookie 17141501922617465952 [ 7583.545611] Lustre: Failing over lustre-MDT0001 [ 7583.550165] LustreError: 151696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7588.832993] LustreError: 218339:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8bb5b0eed880 x1873628476396544/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7588.844129] LustreError: 218339:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7588.979494] Lustre: server umount lustre-MDT0001 complete [ 7604.622832] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7604.625195] LDISKFS-fs (dm-0): recovery complete [ 7604.632382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.639356] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7604.643498] LDISKFS-fs (dm-1): recovery complete [ 7604.652512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.761110] LustreError: 219439:0:(llog.c:1655:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7604.765604] Lustre: 219439:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7608.288537] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5aed42300 x1873628476397696/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7610.322270] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7610.464663] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7614.596837] Lustre: 219453:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5afee6d80 x1873628510580096/t47244640326(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:39/0 lens 560/2880 e 0 to 0 dl 1786838909 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7614.600830] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3628 to 0x280000400:3649) [ 7614.600982] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3628 to 0x2c0000400:3649) [ 7614.624677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13217) [ 7614.624754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13185) [ 7616.877758] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7617.506146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7618.081705] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7622.040097] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 20:07:58 (1786838878) [ 7622.478518] LustreError: 220230:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb48572fb80 x1873628510614144/t51539607622(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:21/0 lens 560/448 e 0 to 0 dl 1786838891 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7622.486864] LustreError: 220230:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 7628.211781] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7628.977565] Lustre: Failing over lustre-MDT0000 [ 7629.142342] Lustre: server umount lustre-MDT0000 complete [ 7638.618046] Lustre: lustre-MDT0001: Client 778185d9-ba67-4d54-8e96-e0aee8a989d9 (at 192.168.203.35@tcp) reconnecting [ 7638.621559] Lustre: Skipped 2 previous similar messages [ 7643.601928] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7643.604079] LDISKFS-fs (dm-0): recovery complete [ 7643.612612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.345253] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7649.278911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13249) [ 7649.278932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13217) [ 7651.326583] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7651.913451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7655.970174] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 20:08:32 (1786838912) [ 7656.377405] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7656.380386] Lustre: Skipped 2 previous similar messages [ 7659.576541] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7660.369121] Lustre: Failing over lustre-MDT0001 [ 7660.472353] Lustre: server umount lustre-MDT0001 complete [ 7675.708109] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7675.710064] LDISKFS-fs (dm-1): recovery complete [ 7675.718668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.607532] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7681.017914] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3670 to 0x2c0000400:3713) [ 7681.018685] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3670 to 0x280000400:3713) [ 7681.024210] Lustre: 220230:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5900aad80 x1873628510646400/t51539607693(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:79/0 lens 560/2880 e 0 to 0 dl 1786838949 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7681.039078] Lustre: 220230:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7683.270101] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7683.964387] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7688.205630] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 20:09:04 (1786838944) [ 7694.969502] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7698.059859] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7698.840117] Lustre: Failing over lustre-MDT0000 [ 7699.097907] Lustre: server umount lustre-MDT0000 complete [ 7705.177663] Lustre: lustre-MDT0001: Client 778185d9-ba67-4d54-8e96-e0aee8a989d9 (at 192.168.203.35@tcp) reconnecting [ 7713.623138] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7713.625433] LDISKFS-fs (dm-0): recovery complete [ 7713.629827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.675296] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7713.678574] LustreError: Skipped 8 previous similar messages [ 7713.748265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.750833] Lustre: Skipped 15 previous similar messages [ 7713.767245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7713.769664] Lustre: Skipped 15 previous similar messages [ 7715.285150] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7715.417267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7715.422308] Lustre: Skipped 14 previous similar messages [ 7718.884435] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7718.890087] Lustre: Skipped 51 previous similar messages [ 7718.909565] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7718.913862] Lustre: Skipped 15 previous similar messages [ 7718.932804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13249) [ 7718.933234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13281) [ 7721.243244] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7721.914742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7724.005708] Lustre: Failing over lustre-MDT0001 [ 7724.106994] Lustre: server umount lustre-MDT0001 complete [ 7738.802560] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7738.809063] LDISKFS-fs (dm-1): recovery complete [ 7738.817450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.575928] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7744.504436] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3745) [ 7744.504464] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3745) [ 7746.745618] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7747.448299] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7751.836139] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 20:10:07 (1786839007) [ 7752.324977] LustreError: 219453:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb5be51ce00 x1873628510715392/t60129542214(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:151/0 lens 560/448 e 0 to 0 dl 1786839021 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7752.333101] LustreError: 219453:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 2 previous similar messages [ 7758.359567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7761.338340] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7762.088927] Lustre: Failing over lustre-MDT0000 [ 7762.272970] Lustre: server umount lustre-MDT0000 complete [ 7763.770662] LustreError: 151697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839021 with bad export cookie 17141501922617479742 [ 7763.772718] Lustre: Failing over lustre-MDT0001 [ 7763.775532] LustreError: 151697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7763.891874] Lustre: server umount lustre-MDT0001 complete [ 7778.958945] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7778.962085] LDISKFS-fs (dm-0): recovery complete [ 7778.968813] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7778.968940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.970447] LDISKFS-fs (dm-1): recovery complete [ 7778.976987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.073062] LustreError: 230594:0:(llog.c:1655:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7779.077032] Lustre: 230594:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7780.320131] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786839022/real 1786839022] req@ffff8bb5b0830700 x1873628476558080/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786839038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7780.329824] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7788.512401] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5900a8700 x1873628476561792/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7790.110650] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7790.187588] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7794.746616] Lustre: 230606:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb5aed40380 x1873628510715392/t60129542214(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:193/0 lens 560/2880 e 0 to 0 dl 1786839063 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7794.750578] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3756 to 0x280000400:3777) [ 7794.750624] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3756 to 0x2c0000400:3777) [ 7794.756875] Lustre: 230606:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7794.769500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13313) [ 7794.769940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13281) [ 7796.896204] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7797.524642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7798.122755] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7802.076023] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 20:10:58 (1786839058) [ 7805.377979] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7806.094160] Lustre: Failing over lustre-MDT0001 [ 7806.098416] LustreError: 232399:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x17:0x0].0x0 (ffff8bb5b4164d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7806.107889] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [ 7806.111034] Lustre: Skipped 26 previous similar messages [ 7812.064713] LustreError: 232399:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8bb5be788380 x1873628476599424/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7812.072686] LustreError: 232399:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7812.158704] Lustre: server umount lustre-MDT0001 complete [ 7826.487533] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7826.488734] LDISKFS-fs (dm-1): recovery complete [ 7826.493668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.106407] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7832.056228] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3809) [ 7832.056586] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3809) [ 7834.139463] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7834.746345] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7838.487167] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 20:11:34 (1786839094) [ 7841.587375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7842.327114] Lustre: Failing over lustre-MDT0000 [ 7842.627728] Lustre: server umount lustre-MDT0000 complete [ 7857.426714] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7857.428982] LDISKFS-fs (dm-0): recovery complete [ 7857.435284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.177849] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7862.789223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13313) [ 7862.792425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13345) [ 7865.226248] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7865.869924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7869.977907] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 20:12:05 (1786839125) [ 7873.523833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7876.600848] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7877.401584] Lustre: Failing over lustre-MDT0000 [ 7877.633032] Lustre: server umount lustre-MDT0000 complete [ 7878.113241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7878.113285] LustreError: 180602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.119951] LustreError: Skipped 13 previous similar messages [ 7878.126558] LustreError: 180602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 215 previous similar messages [ 7892.424707] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7892.426091] LDISKFS-fs (dm-0): recovery complete [ 7892.430905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7894.065498] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7897.602878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13377) [ 7897.602994] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13345) [ 7899.612897] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7900.201907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7902.130792] Lustre: Failing over lustre-MDT0001 [ 7902.210715] Lustre: server umount lustre-MDT0001 complete [ 7916.511619] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7916.513138] LDISKFS-fs (dm-1): recovery complete [ 7916.517122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.051445] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7921.654910] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3841) [ 7921.655172] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3841) [ 7923.617088] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7924.224680] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7927.995592] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 20:13:04 (1786839184) [ 7928.449929] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7928.452451] Lustre: Skipped 5 previous similar messages [ 7931.522136] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7934.456663] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7935.157724] Lustre: Failing over lustre-MDT0000 [ 7935.408842] Lustre: server umount lustre-MDT0000 complete [ 7936.765764] LustreError: 151697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839194 with bad export cookie 17141501922617490186 [ 7936.767239] Lustre: Failing over lustre-MDT0001 [ 7936.770372] LustreError: 151697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7936.992644] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7936.997314] Lustre: Skipped 65 previous similar messages [ 7943.137239] LustreError: 240646:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8bb5b8b49180 x1873628476704768/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7943.147895] LustreError: 240646:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7943.243156] Lustre: server umount lustre-MDT0001 complete [ 7958.605175] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7958.607620] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7958.608045] LDISKFS-fs (dm-0): recovery complete [ 7958.609160] LDISKFS-fs (dm-1): recovery complete [ 7958.615459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.622395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.711890] LustreError: 241744:0:(llog.c:1655:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7958.716328] Lustre: 241744:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7961.571371] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd64a84bf [ 7963.356151] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7963.530757] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7967.879155] Lustre: 241755:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8bb490250e00 x1873628510794880/t73014444034(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:391/0 lens 496/2888 e 0 to 0 dl 1786839261 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7967.880437] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3873) [ 7967.880434] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3873) [ 7967.894340] Lustre: 241755:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7978.121424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13377) [ 7978.121632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13409) [ 7980.600538] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7981.217439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7981.857187] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7985.724351] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 20:14:01 (1786839241) [ 7989.429636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7990.205594] Lustre: Failing over lustre-MDT0000 [ 7990.347908] Lustre: server umount lustre-MDT0000 complete [ 8001.627127] Lustre: lustre-MDT0001: Client 778185d9-ba67-4d54-8e96-e0aee8a989d9 (at 192.168.203.35@tcp) reconnecting [ 8004.841473] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8004.844451] LDISKFS-fs (dm-0): recovery complete [ 8004.848862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.475394] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8010.243087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13441) [ 8010.243251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13409) [ 8012.189642] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8012.795961] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8016.435825] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 20:14:32 (1786839272) [ 8016.829085] LustreError: 241755:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8bb489329500 x1873628510821376/t77309411333(0) o36->778185d9-ba67-4d54-8e96-e0aee8a989d9@192.168.203.35@tcp:415/0 lens 496/456 e 0 to 0 dl 1786839285 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8016.841498] LustreError: 241755:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 5 previous similar messages [ 8019.493538] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8020.151272] Lustre: Failing over lustre-MDT0001 [ 8020.249639] Lustre: server umount lustre-MDT0001 complete [ 8034.551553] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8034.553456] LDISKFS-fs (dm-1): recovery complete [ 8034.558936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.136429] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8039.921751] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3905) [ 8039.921770] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3905) [ 8041.812802] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8042.387909] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8046.020597] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 20:15:02 (1786839302) [ 8049.089606] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8051.825164] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8052.567459] Lustre: Failing over lustre-MDT0000 [ 8052.727617] Lustre: server umount lustre-MDT0000 complete [ 8066.914470] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8066.915939] LDISKFS-fs (dm-0): recovery complete [ 8066.920414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.407387] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8072.194190] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13441) [ 8072.194303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13473) [ 8073.997114] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8074.504946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8076.408856] Lustre: Failing over lustre-MDT0001 [ 8076.485208] Lustre: server umount lustre-MDT0001 complete [ 8090.592710] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8090.593950] LDISKFS-fs (dm-1): recovery complete [ 8090.599039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.060233] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8095.730676] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3937) [ 8095.730732] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3937) [ 8097.569642] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8098.139976] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8101.565094] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 20:15:57 (1786839357) [ 8104.573489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8107.115393] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8107.780215] Lustre: Failing over lustre-MDT0000 [ 8107.908296] Lustre: server umount lustre-MDT0000 complete [ 8109.249729] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839367 with bad export cookie 17141501922617497753 [ 8109.251232] Lustre: Failing over lustre-MDT0001 [ 8109.254422] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8109.361414] Lustre: server umount lustre-MDT0001 complete [ 8124.488261] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8124.490716] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8124.491328] LDISKFS-fs (dm-0): recovery complete [ 8124.492622] LDISKFS-fs (dm-1): recovery complete [ 8124.498891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.501179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.608842] LustreError: 252960:0:(llog.c:1655:llog_backup()) MGC192.168.203.135@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8124.612788] Lustre: 252960:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.135@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8127.968167] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786839369/real 1786839369] req@ffff8bb5bf01e300 x1873628476840192/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786839385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8127.978986] Lustre: 3622:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 8134.114242] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xede2d92dd64aa24e [ 8135.750547] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8135.840031] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8140.341499] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3969) [ 8140.341509] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3969) [ 8140.366993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13473) [ 8140.366997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13505) [ 8142.172927] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8142.688769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8143.206875] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8146.897590] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 20:16:43 (1786839403) [ 8147.532248] Lustre: 254323:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 778185d9-ba67-4d54-8e96-e0aee8a989d9 at adminstrative request [ 8152.479884] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 20:16:48 (1786839408) [ 8155.106535] Lustre: Failing over lustre-MDT0000 [ 8155.217615] Lustre: server umount lustre-MDT0000 complete [ 8168.182045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.636358] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8173.579141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13557 to 0x280000401:13601) [ 8173.579164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13569) [ 8175.361382] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8175.910708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8179.401751] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 20:17:15 (1786839435) [ 8184.178796] Lustre: Failing over lustre-OST0000 [ 8184.242202] Lustre: server umount lustre-OST0000 complete [ 8197.308271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.472763] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8203.502442] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8204.083715] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8207.898234] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 20:17:44 (1786839464) [ 8209.348791] Lustre: Failing over lustre-MDT0000 [ 8209.517115] Lustre: server umount lustre-MDT0000 complete [ 8212.221979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.821181] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8214.607490] Lustre: lustre-MDT0000: Denying connection for new client dc66fd3b-a94a-4884-8c3c-3b91281e8652 (at 192.168.203.35@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8214.618150] Lustre: Skipped 1 previous similar message [ 8217.591663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13557 to 0x280000401:13633) [ 8217.591668] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13601) [ 8222.178851] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 20:17:58 (1786839478) [ 8225.271161] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8226.197055] Lustre: Failing over lustre-OST0000 [ 8226.237410] Lustre: server umount lustre-OST0000 complete [ 8240.552974] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8240.554609] LDISKFS-fs (dm-2): recovery complete [ 8240.559597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8242.502916] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8246.143019] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8246.664917] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8250.318384] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 20:18:26 (1786839506) [ 8253.273119] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8255.192747] Lustre: Failing over lustre-OST0000 [ 8255.230905] Lustre: server umount lustre-OST0000 complete [ 8269.507263] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8269.509359] LDISKFS-fs (dm-2): recovery complete [ 8269.513715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8271.228906] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.35@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:13635 extent [0-4194303]: client csum 684e50b4, server csum bedf5044 [ 8271.811371] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8275.795695] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8276.345515] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8280.028479] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 20:18:56 (1786839536) [ 8282.867793] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8285.591158] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8287.684044] Lustre: Failing over lustre-MDT0000 [ 8287.863832] Lustre: server umount lustre-MDT0000 complete [ 8289.126560] LustreError: 184296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839547 with bad export cookie 17141501922617524626 [ 8289.128635] Lustre: Failing over lustre-OST0000 [ 8289.130706] LustreError: 184296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8289.181322] Lustre: server umount lustre-OST0000 complete [ 8303.599990] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8303.601312] LDISKFS-fs (dm-0): recovery complete [ 8303.604194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.848759] LustreError: 3618:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bb5900a8380 x1873628476990080/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8314.913884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8314.915874] Lustre: Skipped 20 previous similar messages [ 8314.936048] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8314.937849] Lustre: Skipped 20 previous similar messages [ 8315.995218] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8315.997639] Lustre: Skipped 20 previous similar messages [ 8316.250859] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8319.972001] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8319.975641] Lustre: Skipped 68 previous similar messages [ 8320.117133] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8320.119897] Lustre: Skipped 20 previous similar messages [ 8320.135123] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13633) [ 8330.726556] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8330.727993] LDISKFS-fs (dm-2): recovery complete [ 8330.731235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.793462] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8339.769082] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 20:19:55 (1786839595) [ 8344.567212] Lustre: Failing over lustre-OST0000 [ 8344.624670] Lustre: server umount lustre-OST0000 complete [ 8345.869937] Lustre: Failing over lustre-MDT0000 [ 8346.159567] Lustre: server umount lustre-MDT0000 complete [ 8358.972921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.012266] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.016441] LustreError: Skipped 10 previous similar messages [ 8360.423740] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8364.539667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13665) [ 8369.779174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8371.891800] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8372.688965] Lustre: lustre-OST0000: Denying connection for new client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 8377.944728] Lustre: lustre-OST0000: Denying connection for new client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 8388.185094] Lustre: lustre-OST0000: Denying connection for new client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 8388.191860] Lustre: Skipped 1 previous similar message [ 8408.664781] Lustre: lustre-OST0000: Denying connection for new client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 8408.669614] Lustre: Skipped 3 previous similar messages [ 8441.500182] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8441.502450] Lustre: 267850:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client dc66fd3b-a94a-4884-8c3c-3b91281e8652@ [ 8441.505631] Lustre: 267850:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8441.508097] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8441.516942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13684 to 0x280000401:13705) [ 8446.085431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 8453.789666] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 8455.699363] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 20:21:51 (1786839711) [ 8457.101096] Lustre: Failing over lustre-OST0000 [ 8457.136223] Lustre: server umount lustre-OST0000 complete [ 8470.175492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.179802] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8476.067196] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 20:22:12 (1786839732) [ 8477.295924] Lustre: Failing over lustre-OST0000 [ 8477.329129] Lustre: server umount lustre-OST0000 complete [ 8478.297359] LustreError: 8407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8478.302924] LustreError: 8407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 222 previous similar messages [ 8490.129958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.811433] LustreError: 271044:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8491.815172] LustreError: 271044:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 78 previous similar messages [ 8492.015659] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8497.632169] Lustre: *** cfs_fail_loc=715, val=40*** [ 8507.360505] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:25 [ 8507.363697] Lustre: Skipped 1 previous similar message [ 8508.003061] Lustre: lustre-OST0000: Client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 8513.504238] Lustre: *** cfs_fail_loc=715, val=40*** [ 8513.505939] Lustre: Skipped 2 previous similar messages [ 8523.744581] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 8523.748142] Lustre: Skipped 1 previous similar message [ 8529.888178] Lustre: *** cfs_fail_loc=715, val=40*** [ 8529.889256] Lustre: Skipped 2 previous similar messages [ 8531.856090] LustreError: 271044:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8531.858151] LustreError: 271044:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 78 previous similar messages [ 8533.644429] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8534.147425] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8537.314684] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 20:23:13 (1786839793) [ 8538.533925] Lustre: Failing over lustre-MDT0000 [ 8538.593105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.596716] Lustre: Skipped 44 previous similar messages [ 8538.599558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8538.602568] Lustre: Skipped 12 previous similar messages [ 8538.705222] Lustre: server umount lustre-MDT0000 complete [ 8551.367577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.727360] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8556.517670] LustreError: 272595:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8561.120160] Lustre: *** cfs_fail_loc=715, val=80*** [ 8561.122018] Lustre: Skipped 2 previous similar messages [ 8570.456517] Lustre: lustre-MDT0000: Client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 8570.461083] Lustre: Skipped 1 previous similar message [ 8572.896483] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8576.480146] Lustre: *** cfs_fail_loc=715, val=80*** [ 8576.481515] Lustre: Skipped 1 previous similar message [ 8586.840577] Lustre: lustre-MDT0000: Client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 8588.256619] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8592.864166] Lustre: *** cfs_fail_loc=715, val=80*** [ 8592.865692] Lustre: Skipped 1 previous similar message [ 8603.224530] Lustre: lustre-MDT0000: Client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8604.640467] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8609.248112] Lustre: *** cfs_fail_loc=715, val=80*** [ 8609.249779] Lustre: Skipped 1 previous similar message [ 8621.024449] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8634.968921] Lustre: lustre-MDT0000: Client 32195771-e3ca-42cf-b552-5a803caa3f11 (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:22 [ 8634.972476] Lustre: Skipped 1 previous similar message [ 8636.384582] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8636.600162] LustreError: 272595:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8636.607161] Lustre: 272595:0:(ldlm_lib.c:2951:target_recovery_thread()) too long recovery - read logs [ 8636.610785] LustreError: dumping log to /tmp/lustre-log.1786839894.272595 [ 8636.669741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13678 to 0x2c0000401:13697) [ 8636.669749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13719 to 0x280000401:13737) [ 8638.604583] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8639.122773] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8642.554390] Lustre: DEBUG MARKER: == replay-single test complete, duration 8423 sec ======== 20:24:58 (1786839898) [ 8643.056987] Lustre: DEBUG MARKER: === replay-single: start cleanup 20:24:59 (1786839899) === [ 8645.480731] Lustre: DEBUG MARKER: === replay-single: finish cleanup 20:25:01 (1786839901) === [ 8646.173310] Lustre: Failing over lustre-MDT0000 [ 8646.353784] Lustre: server umount lustre-MDT0000 complete [ 8661.629169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8663.119748] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8667.129742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13678 to 0x2c0000401:13729) [ 8667.130062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13719 to 0x280000401:13769) [ 8668.765648] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8669.247236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8677.344544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8677.346475] LustreError: Skipped 16 previous similar messages [ 8681.935567] Lustre: server umount lustre-MDT0000 complete [ 8684.524704] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839942 with bad export cookie 17141501922617556840 [ 8684.528252] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8684.624825] Lustre: server umount lustre-MDT0001 complete [ 8697.864543] Lustre: server umount lustre-OST0000 complete [ 8710.880032] Lustre: server umount lustre-OST0001 complete [ 8715.989879] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 8717.025775] Key type lgssc unregistered [ 8717.165290] LNet: 277417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8717.167959] LNetError: 277417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8717.174313] LNet: Removed LNI 192.168.203.135@tcp [ 8717.521111] Key type .llcrypt unregistered [ 8717.522292] Key type ._llcrypt unregistered