[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489069001 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002406] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007384] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010128] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.021735] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026051] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] *** VALIDATE tmpfs *** [ 0.030450] *** VALIDATE proc *** [ 0.032162] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.040970] debug: unmapping init [mem 0xffffffffa4659000-0xffffffffa4660fff] [ 0.042166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043756] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051297] rcu: Hierarchical SRCU implementation. [ 0.053547] smp: Bringing up secondary CPUs ... [ 0.054616] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.058273] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144024] node 0 deferred pages initialised in 77ms [ 0.147196] devtmpfs: initialized [ 0.148299] x86/mm: Memory block size: 128MB [ 0.151890] gcov: version magic: 0x41383552 [ 0.154182] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161339] pinctrl core: initialized pinctrl subsystem [ 0.163193] [ 0.163774] ************************************************************* [ 0.166014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168011] ** ** [ 0.170017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172013] ** ** [ 0.175018] ** This means that this kernel is built to expose internal ** [ 0.177013] ** IOMMU data structures, which may compromise security on ** [ 0.179013] ** your system. ** [ 0.182016] ** ** [ 0.184012] ** If you see this message and you are not debugging the ** [ 0.186013] ** kernel, report this immediately to your vendor! ** [ 0.189014] ** ** [ 0.191012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193011] ************************************************************* [ 0.194615] NET: Registered protocol family 16 [ 0.196426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205029] cpuidle: using governor menu [ 0.206768] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207594] PCI: Using configuration type 1 for base access [ 0.209147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.219140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225170] cryptd: max_cpu_qlen set to 1000 [ 0.228270] ACPI: Added _OSI(Module Device) [ 0.230018] ACPI: Added _OSI(Processor Device) [ 0.232015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238494] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245363] ACPI: Interpreter enabled [ 0.246000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246000] ACPI: Using IOAPIC for interrupt routing [ 0.247101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.251364] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261478] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263093] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274276] acpiphp: Slot [2] registered [ 0.276129] acpiphp: Slot [3] registered [ 0.277171] acpiphp: Slot [4] registered [ 0.279104] acpiphp: Slot [5] registered [ 0.281164] acpiphp: Slot [6] registered [ 0.282115] acpiphp: Slot [7] registered [ 0.283190] acpiphp: Slot [8] registered [ 0.285112] acpiphp: Slot [9] registered [ 0.286185] acpiphp: Slot [10] registered [ 0.288134] acpiphp: Slot [11] registered [ 0.290081] acpiphp: Slot [12] registered [ 0.292114] acpiphp: Slot [13] registered [ 0.293127] acpiphp: Slot [14] registered [ 0.296191] acpiphp: Slot [15] registered [ 0.298171] acpiphp: Slot [16] registered [ 0.300168] acpiphp: Slot [17] registered [ 0.301152] acpiphp: Slot [18] registered [ 0.303127] acpiphp: Slot [19] registered [ 0.304132] acpiphp: Slot [20] registered [ 0.306120] acpiphp: Slot [21] registered [ 0.307098] acpiphp: Slot [22] registered [ 0.309164] acpiphp: Slot [23] registered [ 0.311160] acpiphp: Slot [24] registered [ 0.313150] acpiphp: Slot [25] registered [ 0.314116] acpiphp: Slot [26] registered [ 0.316119] acpiphp: Slot [27] registered [ 0.318118] acpiphp: Slot [28] registered [ 0.320141] acpiphp: Slot [29] registered [ 0.322169] acpiphp: Slot [30] registered [ 0.324115] acpiphp: Slot [31] registered [ 0.326112] PCI host bridge to bus 0000:00 [ 0.328038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333047] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336045] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339040] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.342042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344283] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348813] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353728] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369546] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380406] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383960] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389971] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408019] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.413000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.451000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480044] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.494064] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518018] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.525000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.531027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.538029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555031] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.565000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.575024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.594000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.599017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.605023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.615019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.625627] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.628451] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.630458] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.633541] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.635270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.640029] iommu: Default domain type: Passthrough [ 0.641000] SCSI subsystem initialized [ 0.642205] ACPI: bus type USB registered [ 0.644190] usbcore: registered new interface driver usbfs [ 0.646126] usbcore: registered new interface driver hub [ 0.649128] usbcore: registered new device driver usb [ 0.651245] pps_core: LinuxPPS API ver. 1 registered [ 0.653018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.656125] PTP clock support registered [ 0.658176] EDAC MC: Ver: 3.0.0 [ 0.661182] PCI: Using ACPI for IRQ routing [ 0.664063] NetLabel: Initializing [ 0.665013] NetLabel: domain hash size = 128 [ 0.667017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.669105] NetLabel: unlabeled traffic allowed by default [ 0.672051] vgaarb: loaded [ 0.674394] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.682423] clocksource: Switched to clocksource kvm-clock [ 0.797692] VFS: Disk quotas dquot_6.6.0 [ 0.799131] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801870] *** VALIDATE ramfs *** [ 0.803167] *** VALIDATE hugetlbfs *** [ 0.804757] pnp: PnP ACPI init [ 0.807238] pnp: PnP ACPI: found 6 devices [ 0.827552] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.830955] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.833212] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.835513] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.837835] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.840085] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.842821] NET: Registered protocol family 2 [ 0.845328] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.850111] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.854763] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.860412] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.863982] TCP: Hash tables configured (established 65536 bind 65536) [ 0.866878] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.869907] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.872581] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875494] NET: Registered protocol family 1 [ 0.878563] RPC: Registered named UNIX socket transport module. [ 0.880673] RPC: Registered udp transport module. [ 0.882286] RPC: Registered tcp transport module. [ 0.883832] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.886101] NET: Registered protocol family 44 [ 0.887748] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.889888] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.891897] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.894155] PCI: CLS 0 bytes, default 64 [ 0.895890] Unpacking initramfs... [ 2.314761] debug: unmapping init [mem 0xffff8a1d7cc54000-0xffff8a1d7ffbffff] [ 2.319103] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.321386] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.324129] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.836536] Initialise system trusted keyrings [ 2.838343] Key type blacklist registered [ 2.840193] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.850227] zbud: loaded [ 2.853514] *** VALIDATE nfs *** [ 2.854629] *** VALIDATE nfs4 *** [ 2.855862] pstore: using deflate compression [ 2.858993] Platform Keyring initialized [ 2.959111] NET: Registered protocol family 38 [ 2.960676] Key type asymmetric registered [ 2.962194] Asymmetric key parser 'x509' registered [ 2.964214] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.968129] io scheduler mq-deadline registered [ 2.969670] io scheduler kyber registered [ 2.970994] io scheduler bfq registered [ 2.973898] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.977665] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.980397] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.983621] ACPI: Power Button [PWRF] [ 3.075350] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.167732] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.348952] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.439775] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.632576] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.671027] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.707365] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.713342] Non-volatile memory driver v1.3 [ 3.715099] Linux agpgart interface v0.103 [ 3.749214] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.751894] vda: detected capacity change from 0 to 68526080 [ 3.765922] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.769367] vdb: detected capacity change from 0 to 1073741824 [ 3.787542] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.791237] vdc: detected capacity change from 0 to 2621440000 [ 3.804639] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.807667] vdd: detected capacity change from 0 to 2621440000 [ 3.822380] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.825181] vde: detected capacity change from 0 to 4294967296 [ 3.838362] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.841375] vdf: detected capacity change from 0 to 4294967296 [ 3.848583] libphy: Fixed MDIO Bus: probed [ 3.857779] usbcore: registered new interface driver usbserial_generic [ 3.860560] usbserial: USB Serial support registered for generic [ 3.863504] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.868135] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.870084] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.872760] mousedev: PS/2 mouse device common for all mice [ 3.875570] rtc_cmos 00:05: RTC can wake from S4 [ 3.877895] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.880440] rtc_cmos 00:05: registered as rtc0 [ 3.885313] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.889235] intel_pstate: CPU model not supported [ 3.893195] hid: raw HID events driver (C) Jiri Kosina [ 3.895939] usbcore: registered new interface driver usbhid [ 3.897804] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.902043] usbhid: USB HID core driver [ 3.902213] drop_monitor: Initializing network drop monitor service [ 3.909656] Initializing XFRM netlink socket [ 3.910905] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.913610] NET: Registered protocol family 10 [ 3.916751] Segment Routing with IPv6 [ 3.918755] NET: Registered protocol family 17 [ 3.920621] mpls_gso: MPLS GSO support [ 3.925633] RAS: Correctable Errors collector initialized. [ 3.927819] AVX version of gcm_enc/dec engaged. [ 3.929349] AES CTR mode by8 optimization enabled [ 4.001432] sched_clock: Marking stable (4001408300, 0)->(4958713970, -957305670) [ 4.005069] registered taskstats version 1 [ 4.007467] Loading compiled-in X.509 certificates [ 4.010462] zswap: loaded using pool lzo/zbud [ 4.035638] Key type big_key registered [ 4.046903] Key type encrypted registered [ 4.048661] ima: No TPM chip found, activating TPM-bypass! [ 4.051124] ima: Allocated hash algorithm: sha1 [ 4.052733] ima: No architecture policies found [ 4.054282] evm: Initialising EVM extended attributes: [ 4.055779] evm: security.selinux [ 4.057068] evm: security.ima [ 4.058200] evm: security.capability [ 4.059765] evm: HMAC attrs: 0x1 [ 4.062794] rtc_cmos 00:05: setting system clock to 2025-11-17 03:11:49 UTC (1763349109) [ 4.069687] debug: unmapping init [mem 0xffffffffa5603000-0xffffffffa57fffff] [ 4.072674] debug: unmapping init [mem 0xffffffffa4382000-0xffffffffa4658fff] [ 4.080347] Write protecting the kernel read-only data: 28672k [ 4.083080] debug: unmapping init [mem 0xffffffffa2a03000-0xffffffffa2bfffff] [ 4.085726] debug: unmapping init [mem 0xffffffffa3314000-0xffffffffa33fffff] [ 4.117856] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.125536] systemd[1]: Detected virtualization kvm. [ 4.126881] systemd[1]: Detected architecture x86-64. [ 4.128714] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.157068] systemd[1]: No hostname configured. [ 4.158390] systemd[1]: Set hostname to . [ 4.160334] random: systemd: uninitialized urandom read (16 bytes read) [ 4.162548] systemd[1]: Initializing machine ID from random generator. [ 4.197869] random: ln: uninitialized urandom read (6 bytes read) [ 4.282944] random: systemd: uninitialized urandom read (16 bytes read) [ 4.285437] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.289800] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.294863] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.895528] device-mapper: uevent: version 1.0.3 [ 4.897684] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.573349] virtio_net virtio0 ens2: renamed from eth0 [ 5.617428] scsi host0: ata_piix [ 5.618172] random: fast init done [ 5.668136] scsi host1: ata_piix [ 5.669881] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.671464] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.254174] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.574596] random: crng init done [ 10.575887] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.871842] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.040528] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.313936] SELinux: Disabled at runtime. [ 12.373427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.382732] systemd[1]: Detected virtualization kvm. [ 12.384420] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.869410] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.872555] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.878837] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.882326] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.889466] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.898540] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.905046] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slic[ 13.033446] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.293919] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.598591] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.628312] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.739635] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.758496] EDAC sbridge: Ver: 1.1.2 [ 15.443878] Key type dns_resolver registered [ 16.093997] NFS: Registering the id_resolver key type [ 16.097449] Key type id_resolver registered [ 16.102519] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. [ 25.300654] hrtimer: interrupt took 3055258 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg231-server login: [ 67.397229] libcfs: loading out-of-tree module taints kernel. [ 67.413248] Key type ._llcrypt registered [ 67.415365] Key type .llcrypt registered [ 67.501762] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_hostid [ 83.160951] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 84.388626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.407675] alg: No test for adler32 (adler32-zlib) [ 85.736938] Lustre: Lustre: Build Version: 2.16.61_42_gb9e4aef [ 86.513356] LNet: Added LNI 192.168.202.131@tcp [8/256/0/180] [ 88.264178] Key type lgssc registered [ 89.716109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.305251] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.222826] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.171048] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.128450] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.165580] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 141.430844] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing load_modules_local [ 152.907839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.967841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.982624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.208090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.241184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.315592] Lustre: lustre-MDT0000: new disk, initializing [ 154.438893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.453954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.118365] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.310677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.380390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 170.455950] Lustre: 6513:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.498070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.502558] Lustre: Skipped 1 previous similar message [ 170.576274] Lustre: lustre-MDT0001: new disk, initializing [ 170.667537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.702026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.714267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 173.978091] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.320480] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.381303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.481456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.809414] Lustre: lustre-OST0000: new disk, initializing [ 187.822354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 187.901161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.347277] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.558851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.575471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.650136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.043390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.110454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.179547] Lustre: lustre-OST0001: new disk, initializing [ 205.182435] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.224981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.220181] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.521656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 212.535675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 212.676196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.077401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.680417] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.583115] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.337181] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing yml_node [ 240.762597] Lustre: DEBUG MARKER: Client: 2.16.61.42 [ 242.717258] Lustre: DEBUG MARKER: MDS: 2.16.61.42 [ 244.675678] Lustre: DEBUG MARKER: OSS: 2.16.61.42 [ 246.061054] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sun Nov 16 22:15:49 EST 2025 [ 260.209510] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 262.041253] Lustre: DEBUG MARKER: === replay-single: start setup 22:16:05 (1763349365) === [ 264.923684] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing check_config_client /mnt/lustre [ 283.800922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.135386] Lustre: 13193:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 290.084239] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.501302] Lustre: DEBUG MARKER: === replay-single: finish setup 22:16:38 (1763349398) === [ 297.444549] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 22:16:40 (1763349400) [ 304.489727] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 306.168843] Lustre: Failing over lustre-MDT0000 [ 306.322938] LustreError: 14064:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 306.453055] Lustre: server umount lustre-MDT0000 complete [ 309.217735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.225298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.734685] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.759273] Lustre: Skipped 1 previous similar message [ 312.529660] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.547809] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 314.849848] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.876601] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 317.655594] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.972250] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.000773] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 325.088961] LustreError: 8617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.096764] LustreError: 8617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 326.107566] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349415/real 1763349415] req@ffff8a1dff7af800 x1849005641409152/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 326.152365] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.963716] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 327.965703] LDISKFS-fs (dm-0): recovery complete [ 327.983160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.038432] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.064655] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 336.359454] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1cc3248000 x1849005641417600/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 336.579850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.091159] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 340.256538] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 342.010669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 342.087591] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 346.545427] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 347.875800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 354.882447] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 22:17:38 (1763349458) [ 356.442041] Lustre: Failing over lustre-OST0000 [ 356.475147] LustreError: 15497:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 356.477155] LustreError: 15497:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 356.510063] Lustre: server umount lustre-OST0000 complete [ 357.347730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 357.352872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.358820] Lustre: Skipped 1 previous similar message [ 357.362363] LustreError: 13487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.373319] LustreError: 13487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 372.340082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 372.473830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.951781] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 374.347106] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 374.347685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 374.357431] Lustre: Skipped 3 previous similar messages [ 376.744188] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 382.333647] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 383.564363] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 391.412449] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 22:18:15 (1763349495) [ 398.040517] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 399.965410] Lustre: Failing over lustre-MDT0000 [ 400.075083] LustreError: 17204:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 400.082649] LustreError: 17204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 400.171907] Lustre: server umount lustre-MDT0000 complete [ 403.425538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 403.425807] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.429920] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.429931] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 403.465031] Lustre: Skipped 3 previous similar messages [ 418.786470] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349508/real 1763349508] req@ffff8a1dffa8f100 x1849005641464192/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 418.807916] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.202574] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 421.204631] LDISKFS-fs (dm-0): recovery complete [ 421.213020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.028365] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1dffa4d180 x1849005641471872/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 429.299901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 433.870033] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 434.656933] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 434.663799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 434.671271] Lustre: Skipped 1 previous similar message [ 436.513419] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 441.557548] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 446.677547] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 451.798108] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 456.916929] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 467.155507] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:36 [ 467.174668] Lustre: Skipped 1 previous similar message [ 487.636585] Lustre: lustre-MDT0000: Denying connection for new client 0a414fdb-8602-430b-ac87-925922dcb942 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 487.666202] Lustre: Skipped 3 previous similar messages [ 504.000304] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 504.013519] Lustre: 17857:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 44ae346c-0ce4-4f78-ae50-a634e0037614@ [ 504.027724] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 504.045189] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 504.045417] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 504.051762] Lustre: Skipped 2 previous similar messages [ 504.066251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 504.073036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 517.896746] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 22:20:20 (1763349620) [ 525.843361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 528.076806] Lustre: Failing over lustre-MDT0000 [ 528.247300] LustreError: 18914:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 528.259089] LustreError: 18914:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 528.356174] Lustre: server umount lustre-MDT0000 complete [ 529.888464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 529.905160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.940993] Lustre: Skipped 1 previous similar message [ 529.953301] LustreError: 8617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.982796] LustreError: 8617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 548.320130] Lustre: 3656:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349637/real 1763349637] req@ffff8a1dffa8ea00 x1849005641524864/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 548.364845] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.283889] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 550.285939] LDISKFS-fs (dm-0): recovery complete [ 550.307434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.561988] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b87a4dd [ 558.590779] Lustre: MGC192.168.202.131@tcp: Connection restored to 0@lo (at 0@lo) [ 558.799433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 562.846546] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 564.199788] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 565.401630] Lustre: lustre-MDT0000: Denying connection for new client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 565.424513] Lustre: Skipped 3 previous similar messages [ 632.018808] Lustre: lustre-MDT0000: Denying connection for new client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:01 [ 632.042656] Lustre: Skipped 13 previous similar messages [ 634.002117] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 634.004162] Lustre: 19570:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0a414fdb-8602-430b-ac87-925922dcb942@ [ 634.030891] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 634.042876] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 634.042945] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 634.049443] Lustre: Skipped 3 previous similar messages [ 634.074984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 634.079742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 647.413680] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 22:22:30 (1763349750) [ 655.181772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 657.713843] Lustre: Failing over lustre-MDT0000 [ 657.857563] LustreError: 20650:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 657.911364] LustreError: 20650:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 658.013883] Lustre: server umount lustre-MDT0000 complete [ 659.940434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 659.949864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.963080] Lustre: Skipped 3 previous similar messages [ 659.969827] LustreError: 8617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.976340] LustreError: 8617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 676.833983] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349766/real 1763349766] req@ffff8a1cc3248e00 x1849005641582336/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 676.911233] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.371895] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 681.374993] LDISKFS-fs (dm-0): recovery complete [ 681.383745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.075650] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1dc49a3100 x1849005641590272/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 687.393672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.432070] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 688.350572] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 691.340183] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 692.706631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 692.825090] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 692.886521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 692.888407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 698.345797] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 700.022618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 708.215917] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 22:23:31 (1763349811) [ 715.343965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 717.074062] Lustre: Failing over lustre-MDT0000 [ 717.200104] LustreError: 22478:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 717.211611] LustreError: 22478:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 717.402807] Lustre: server umount lustre-MDT0000 complete [ 718.308564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.331885] Lustre: Skipped 5 previous similar messages [ 734.160091] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349823/real 1763349823] req@ffff8a1cc304dc00 x1849005641612672/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 734.185467] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.258917] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 739.261049] LDISKFS-fs (dm-0): recovery complete [ 739.277748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.419627] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1ded1ece00 x1849005641620992/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 744.647595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 744.677829] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 748.067822] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 750.053070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.073281] Lustre: Skipped 3 previous similar messages [ 750.148804] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 750.182644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 750.187374] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 754.563614] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 756.086607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 764.572187] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 22:24:27 (1763349867) [ 772.856796] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 775.066060] Lustre: Failing over lustre-MDT0000 [ 775.222533] LustreError: 24304:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 775.226470] LustreError: 24304:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 775.301544] Lustre: server umount lustre-MDT0000 complete [ 775.656434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 775.671144] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.686776] Lustre: Skipped 3 previous similar messages [ 796.624161] Lustre: 3656:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349886/real 1763349886] req@ffff8a1cc1cb1c00 x1849005641647232/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 796.651618] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 796.872639] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 796.877709] LDISKFS-fs (dm-0): recovery complete [ 796.892872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.173084] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 808.334148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 810.899201] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 812.670039] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 812.759398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 812.759519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 818.159910] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 820.261977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 830.011528] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 22:25:33 (1763349933) [ 837.649279] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 839.662432] Lustre: Failing over lustre-MDT0000 [ 839.836289] LustreError: 26131:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 839.839734] LustreError: 26131:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 839.937928] Lustre: server umount lustre-MDT0000 complete [ 843.232756] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.245336] Lustre: Skipped 4 previous similar messages [ 859.620754] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349948/real 1763349948] req@ffff8a1ddf6b7480 x1849005641681664/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 859.660525] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 862.096215] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 862.098241] LDISKFS-fs (dm-0): recovery complete [ 862.119098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.089511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.092668] Lustre: Skipped 2 previous similar messages [ 870.142218] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 871.272553] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 873.765311] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 875.492262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 875.500318] Lustre: Skipped 7 previous similar messages [ 875.580260] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 875.617028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 875.621423] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 880.441631] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 881.812368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 889.534856] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 22:26:33 (1763349993) [ 895.906308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 897.531230] Lustre: Failing over lustre-MDT0000 [ 897.772306] Lustre: server umount lustre-MDT0000 complete [ 901.090865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 916.451629] LustreError: 10294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 916.464206] LustreError: 10294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 150 previous similar messages [ 917.473150] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350006/real 1763350006] req@ffff8a1cc382b800 x1849005641716352/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763350022 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 917.519378] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.296541] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 922.299267] LDISKFS-fs (dm-0): recovery complete [ 922.309418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.917807] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 930.708042] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 932.427511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 932.431810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 938.744528] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 940.826980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 950.758533] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 22:27:33 (1763350053) [ 952.235703] Lustre: *** cfs_fail_loc=13b, val=315*** [ 952.237588] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 952.239442] LustreError: 6523:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1ded1ee300 x1849005625720960/t38654705666(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:3/0 lens 392/456 e 0 to 0 dl 1763350068 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 961.397489] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 963.931523] Lustre: Failing over lustre-MDT0000 [ 964.133031] LustreError: 29837:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 964.164910] LustreError: 29837:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 964.286045] Lustre: server umount lustre-MDT0000 complete [ 968.169818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 984.528522] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350073/real 1763350073] req@ffff8a1cc382bb80 x1849005641754880/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763350089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 984.551792] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.825915] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 986.827345] LDISKFS-fs (dm-0): recovery complete [ 986.839033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.761665] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1cc3a79c00 x1849005641763328/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 994.137603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 994.243345] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 994.249422] Lustre: Skipped 1 previous similar message [ 998.118354] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 999.469294] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 999.471821] Lustre: Skipped 1 previous similar message [ 999.482469] Lustre: 6524:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dc498a680 x1849005625720960/t38654705666(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:50/0 lens 392/456 e 0 to 0 dl 1763350115 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 999.514539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 999.516950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1005.683420] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1007.633221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1016.524025] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 22:28:39 (1763350119) [ 1023.663845] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1025.618793] Lustre: Failing over lustre-MDT0000 [ 1025.922196] Lustre: server umount lustre-MDT0000 complete [ 1030.114080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.126294] Lustre: Skipped 11 previous similar messages [ 1047.605056] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1047.607734] LDISKFS-fs (dm-0): recovery complete [ 1047.620581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.714249] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b87c79e [ 1055.735668] Lustre: MGC192.168.202.131@tcp: Connection restored to 0@lo (at 0@lo) [ 1055.738397] Lustre: Skipped 11 previous similar messages [ 1056.013106] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1060.068293] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1061.487991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1061.491291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1067.635719] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1069.405946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1077.363315] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 22:29:40 (1763350180) [ 1085.068701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1085.933860] Lustre: *** cfs_fail_loc=114, val=0*** [ 1088.315762] Lustre: Failing over lustre-MDT0000 [ 1088.538570] Lustre: server umount lustre-MDT0000 complete [ 1092.065411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1092.079416] LustreError: Skipped 1 previous similar message [ 1111.088547] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1111.090588] LDISKFS-fs (dm-0): recovery complete [ 1111.112839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.041498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1123.076563] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1123.392559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1123.407461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1131.271952] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1133.213396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1141.758586] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 22:30:45 (1763350245) [ 1149.205607] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1150.263010] Lustre: *** cfs_fail_loc=128, val=0*** [ 1153.113308] Lustre: Failing over lustre-MDT0000 [ 1153.314494] LustreError: 35512:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1153.338730] LustreError: 35512:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 1153.525253] Lustre: server umount lustre-MDT0000 complete [ 1170.401638] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350259/real 1763350259] req@ffff8a1df4c2d180 x1849005641862144/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763350275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1170.428664] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1170.440161] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.456294] LustreError: Skipped 2 previous similar messages [ 1175.587124] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1175.589212] LDISKFS-fs (dm-0): recovery complete [ 1175.600576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.621284] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1df4f41c00 x1849005641870336/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1179.885790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1179.889603] Lustre: Skipped 4 previous similar messages [ 1180.896865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1180.923321] Lustre: Skipped 2 previous similar messages [ 1183.679812] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1185.314850] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1185.328774] Lustre: Skipped 2 previous similar messages [ 1185.373100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1185.373331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1190.578859] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1192.250434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1200.514377] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 22:31:44 (1763350304) [ 1207.906856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1210.196061] Lustre: Failing over lustre-MDT0000 [ 1210.416294] Lustre: server umount lustre-MDT0000 complete [ 1231.114116] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1231.116946] LDISKFS-fs (dm-0): recovery complete [ 1231.128050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.344460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1231.502092] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1231.527763] Lustre: Skipped 1 previous similar message [ 1234.804124] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1236.900154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1236.912713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1242.693796] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1244.266360] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1255.030252] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 22:32:38 (1763350358) [ 1264.462983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1267.023291] Lustre: Failing over lustre-MDT0000 [ 1267.171627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1267.189696] Lustre: Skipped 4 previous similar messages [ 1267.392799] Lustre: server umount lustre-MDT0000 complete [ 1287.480531] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1287.482827] LDISKFS-fs (dm-0): recovery complete [ 1287.491272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.653832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1287.660795] Lustre: Skipped 5 previous similar messages [ 1291.166432] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1293.485261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1293.487406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1298.011262] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1299.582744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1307.489341] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 22:33:30 (1763350410) [ 1313.734836] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1321.843427] Lustre: Failing over lustre-MDT0000 [ 1322.245699] Lustre: server umount lustre-MDT0000 complete [ 1324.003512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.009648] Lustre: Skipped 18 previous similar messages [ 1343.635210] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1343.637196] LDISKFS-fs (dm-0): recovery complete [ 1343.649985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.355913] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1355.268563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1355.271584] Lustre: Skipped 20 previous similar messages [ 1358.213253] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1358.213492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1361.541963] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1363.222092] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1386.138924] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 22:34:49 (1763350489) [ 1393.513402] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1395.182386] Lustre: Failing over lustre-MDT0000 [ 1395.360767] Lustre: server umount lustre-MDT0000 complete [ 1415.812804] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1415.819250] LDISKFS-fs (dm-0): recovery complete [ 1415.842586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.818081] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1dff7adf80 x1849005642070656/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1423.098279] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1423.118746] Lustre: Skipped 2 previous similar messages [ 1426.283944] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1428.561657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1428.565508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1432.540448] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1434.538968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1445.067200] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 22:35:48 (1763350548) [ 1452.822905] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1454.739786] Lustre: Failing over lustre-MDT0000 [ 1454.870509] LustreError: 44683:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1454.873892] LustreError: 44683:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 1455.006940] Lustre: server umount lustre-MDT0000 complete [ 1455.321811] LustreError: 10294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.328207] LustreError: 10294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 257 previous similar messages [ 1459.168577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1459.183778] LustreError: Skipped 2 previous similar messages [ 1474.450867] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1474.453112] LDISKFS-fs (dm-0): recovery complete [ 1474.462357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.543305] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.549640] LustreError: Skipped 4 previous similar messages [ 1476.826187] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1476.829252] Lustre: Skipped 4 previous similar messages [ 1478.120763] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1480.212812] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1480.223077] Lustre: Skipped 4 previous similar messages [ 1480.246265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1480.246749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1484.081965] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1485.606355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1493.353638] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 22:36:36 (1763350596) [ 1500.699378] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1502.466638] Lustre: Failing over lustre-MDT0000 [ 1502.678791] Lustre: server umount lustre-MDT0000 complete [ 1522.529765] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350611/real 1763350611] req@ffff8a1dff7aca80 x1849005642147712/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763350627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1522.560611] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1523.731114] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1523.732704] LDISKFS-fs (dm-0): recovery complete [ 1523.740943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.901648] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b88886b [ 1537.103513] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1538.643881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1538.644820] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1545.153567] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1546.632894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1554.977513] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 22:37:38 (1763350658) [ 1562.380162] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1564.221454] Lustre: Failing over lustre-MDT0000 [ 1564.384762] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1564.553452] Lustre: server umount lustre-MDT0000 complete [ 1585.134169] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1585.136678] LDISKFS-fs (dm-0): recovery complete [ 1585.143869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.672968] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1590.870033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1590.870610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1594.876721] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.575317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1604.193785] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 22:38:27 (1763350707) [ 1611.474209] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1613.144117] Lustre: Failing over lustre-MDT0000 [ 1613.307536] Lustre: server umount lustre-MDT0000 complete [ 1633.234588] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1633.239506] LDISKFS-fs (dm-0): recovery complete [ 1633.247875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.649867] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1648.723566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1648.723644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1652.404470] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1653.609148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1660.360229] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 22:39:23 (1763350763) [ 1667.347180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1669.657054] Lustre: Failing over lustre-MDT0000 [ 1669.860795] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1671.974905] Lustre: server umount lustre-MDT0000 complete [ 1693.350147] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1693.351682] LDISKFS-fs (dm-0): recovery complete [ 1693.357515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.809646] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1ddf707100 x1849005642256896/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1699.824126] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 1699.991640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.994630] Lustre: Skipped 8 previous similar messages [ 1700.017870] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1700.023380] Lustre: Skipped 4 previous similar messages [ 1703.200126] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1705.564571] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1705.565712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1710.061857] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1711.352207] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1718.393525] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 22:40:21 (1763350821) [ 1725.163910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1726.494791] Lustre: Failing over lustre-MDT0000 [ 1726.649826] Lustre: server umount lustre-MDT0000 complete [ 1731.041676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1731.059067] LustreError: Skipped 4 previous similar messages [ 1746.448631] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1746.450923] LDISKFS-fs (dm-0): recovery complete [ 1746.460962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.645396] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (not set up) [ 1750.508620] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1752.192422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1752.192434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1757.975882] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1759.630870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1767.541459] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 22:41:11 (1763350871) [ 1774.442254] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1776.226744] Lustre: Failing over lustre-MDT0000 [ 1776.489453] Lustre: server umount lustre-MDT0000 complete [ 1797.462903] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1797.464907] LDISKFS-fs (dm-0): recovery complete [ 1797.475336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.415172] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1809.997480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1809.998707] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1813.500354] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.874836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1822.018750] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 22:42:05 (1763350925) [ 1828.442987] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1830.113538] Lustre: Failing over lustre-MDT0000 [ 1830.405154] Lustre: server umount lustre-MDT0000 complete [ 1850.319819] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1850.321768] LDISKFS-fs (dm-0): recovery complete [ 1850.328977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.313814] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1856.078847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1856.082141] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1860.471539] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1863.011857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1870.903551] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 22:42:54 (1763350974) [ 1877.982465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1880.022620] Lustre: Failing over lustre-MDT0000 [ 1880.312982] Lustre: server umount lustre-MDT0000 complete [ 1881.572819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.602410] Lustre: Skipped 37 previous similar messages [ 1901.189698] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1901.191549] LDISKFS-fs (dm-0): recovery complete [ 1901.202440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.681855] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1df1652680 x1849005642381312/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1910.972562] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1913.314150] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1913.318166] Lustre: Skipped 40 previous similar messages [ 1913.430512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 1913.432709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1916.698839] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1917.930475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1923.781983] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 22:43:47 (1763351027) [ 1929.201663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1930.849561] Lustre: Failing over lustre-MDT0000 [ 1930.968753] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 1931.092964] Lustre: server umount lustre-MDT0000 complete [ 1949.669563] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1949.671434] LDISKFS-fs (dm-0): recovery complete [ 1949.686751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1952.910498] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1955.417194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1955.417195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1958.646991] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1959.820422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1966.692090] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 22:44:30 (1763351070) [ 1972.619484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1974.325631] Lustre: Failing over lustre-MDT0000 [ 1974.438252] LustreError: 62929:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1974.444952] LustreError: 62929:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 1974.556510] Lustre: server umount lustre-MDT0000 complete [ 1992.173109] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1992.184584] LustreError: Skipped 9 previous similar messages [ 1994.196568] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1994.198825] LDISKFS-fs (dm-0): recovery complete [ 1994.212642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.650906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2002.661961] Lustre: Skipped 9 previous similar messages [ 2005.606150] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2008.082959] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 2008.092387] Lustre: Skipped 9 previous similar messages [ 2008.129656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2008.129766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2011.656757] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2012.750442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2019.211233] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 22:45:22 (1763351122) [ 2024.592481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2025.889604] Lustre: Failing over lustre-MDT0000 [ 2026.036323] Lustre: server umount lustre-MDT0000 complete [ 2043.808146] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351133/real 1763351133] req@ffff8a1ded335500 x1849005642461440/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763351149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2043.826417] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 2044.357969] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2044.360053] LDISKFS-fs (dm-0): recovery complete [ 2044.373377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.114547] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b88bedc [ 2057.395904] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2059.825459] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2059.828192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2064.517980] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2066.206583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2074.495727] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 22:46:17 (1763351177) [ 2081.861379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2083.923750] Lustre: Failing over lustre-MDT0000 [ 2084.150390] Lustre: server umount lustre-MDT0000 complete [ 2085.356399] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.384174] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 330 previous similar messages [ 2104.869611] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2104.872216] LDISKFS-fs (dm-0): recovery complete [ 2104.880386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.969106] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1ddf53e680 x1849005642501632/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2114.887382] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2117.729660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2117.731856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2121.326566] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2122.787393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2131.155659] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 22:47:14 (1763351234) [ 2137.424143] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2139.170938] Lustre: Failing over lustre-MDT0000 [ 2139.406793] Lustre: server umount lustre-MDT0000 complete [ 2159.942670] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2159.944824] LDISKFS-fs (dm-0): recovery complete [ 2159.954824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.913706] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2175.635186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2175.637550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2180.861853] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2182.309523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2189.293789] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 22:48:12 (1763351292) [ 2196.077620] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2197.795586] Lustre: Failing over lustre-MDT0000 [ 2198.104275] Lustre: server umount lustre-MDT0000 complete [ 2218.686888] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2218.690189] LDISKFS-fs (dm-0): recovery complete [ 2218.698401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.465420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2227.471641] Lustre: Skipped 9 previous similar messages [ 2231.129399] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2232.910167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2232.910170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2237.704914] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2239.406321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2247.188845] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 22:49:10 (1763351350) [ 2250.433515] Lustre: 71710:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 61808989-3964-40c4-81ec-dd31f403bbd4 at adminstrative request [ 2256.026129] Lustre: Failing over lustre-MDT0000 [ 2256.224520] Lustre: server umount lustre-MDT0000 complete [ 2258.408307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2258.411972] LustreError: Skipped 3 previous similar messages [ 2272.234641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.443581] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2277.917674] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2277.928403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2282.575531] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2284.405099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2288.812605] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2296.686570] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2303.544722] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 22:50:06 (1763351406) [ 2304.541426] Lustre: 73557:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 61808989-3964-40c4-81ec-dd31f403bbd4 at adminstrative request [ 2312.872517] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 22:50:16 (1763351416) [ 2320.371673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2322.656219] Lustre: Failing over lustre-MDT0000 [ 2322.993543] Lustre: server umount lustre-MDT0000 complete [ 2344.315107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2344.318583] LDISKFS-fs (dm-0): recovery complete [ 2344.326932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.769232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.772681] Lustre: Skipped 11 previous similar messages [ 2353.733054] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2356.393620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2356.395344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2360.152358] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2361.737151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2369.079089] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 22:51:12 (1763351472) [ 2376.150978] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2377.908588] Lustre: Failing over lustre-MDT0000 [ 2378.109689] Lustre: server umount lustre-MDT0000 complete [ 2399.594914] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2399.596865] LDISKFS-fs (dm-0): recovery complete [ 2399.607663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.421331] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1ded5ac000 x1849005642676608/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2408.449651] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 8 previous similar messages [ 2411.996153] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2414.134235] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2414.137795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2418.784676] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2420.189684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2427.230946] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 22:52:10 (1763351530) [ 2432.918378] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2434.552245] Lustre: Failing over lustre-MDT0000 [ 2434.808951] Lustre: server umount lustre-MDT0000 complete [ 2455.606325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2455.612128] LDISKFS-fs (dm-0): recovery complete [ 2455.627566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.137425] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2461.230095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2461.230105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2465.232701] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2466.457576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2473.127156] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 22:52:56 (1763351576) [ 2478.692663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2480.310379] Lustre: Failing over lustre-MDT0000 [ 2480.490360] Lustre: server umount lustre-MDT0000 complete [ 2481.635857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.642442] Lustre: Skipped 43 previous similar messages [ 2501.029231] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2501.031452] LDISKFS-fs (dm-0): recovery complete [ 2501.040031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.750454] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b88eb60 [ 2510.442978] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2513.383355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2513.388075] Lustre: Skipped 45 previous similar messages [ 2513.439748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2513.441071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2516.314841] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2517.642793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2525.061906] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 22:53:48 (1763351628) [ 2531.012932] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2532.388609] Lustre: Failing over lustre-MDT0000 [ 2532.609505] Lustre: server umount lustre-MDT0000 complete [ 2552.834920] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2552.837250] LDISKFS-fs (dm-0): recovery complete [ 2552.846805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.977286] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b88f0c3 [ 2563.798496] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2565.670894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2565.672718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2570.270079] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2571.849765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2579.299664] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 22:54:42 (1763351682) [ 2586.464486] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2588.379049] Lustre: Failing over lustre-MDT0000 [ 2588.514540] LustreError: 83373:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2588.516855] LustreError: 83373:0:(obd_class.h:479:obd_check_dev()) Skipped 87 previous similar messages [ 2588.584657] Lustre: server umount lustre-MDT0000 complete [ 2607.584249] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2607.594302] LustreError: Skipped 10 previous similar messages [ 2609.276950] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2609.278977] LDISKFS-fs (dm-0): recovery complete [ 2609.284769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.832431] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b88f65e [ 2619.905651] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2619.911395] Lustre: Skipped 10 previous similar messages [ 2621.579760] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2623.586851] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2623.598746] Lustre: Skipped 10 previous similar messages [ 2623.626221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2623.627869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2628.674237] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2630.254667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2638.391325] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 22:55:41 (1763351741) [ 2645.279347] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2647.289386] Lustre: Failing over lustre-MDT0000 [ 2647.605818] Lustre: server umount lustre-MDT0000 complete [ 2665.440823] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763351754/real 1763351754] req@ffff8a1dee495c00 x1849005642825984/t0(0) o400->MGC192.168.202.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763351770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2665.464721] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2667.718888] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2667.727236] LDISKFS-fs (dm-0): recovery complete [ 2667.740033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.106542] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2680.403802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2680.403802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2684.939227] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2686.302131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2693.380737] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 22:56:36 (1763351796) [ 2700.670594] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2702.304512] Lustre: Failing over lustre-MDT0000 [ 2702.598279] Lustre: server umount lustre-MDT0000 complete [ 2705.890157] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.907949] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 336 previous similar messages [ 2722.148840] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2722.152684] LDISKFS-fs (dm-0): recovery complete [ 2722.161046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.240945] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2728.011220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2728.014889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2730.999449] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.231335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2739.180536] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 22:57:22 (1763351842) [ 2744.728801] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2746.141345] Lustre: Failing over lustre-MDT0000 [ 2746.352584] Lustre: server umount lustre-MDT0000 complete [ 2766.493996] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2766.498157] LDISKFS-fs (dm-0): recovery complete [ 2766.511368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.842952] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2779.687948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2779.688111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2782.040376] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2783.284193] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2790.221198] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 22:58:13 (1763351893) [ 2795.658828] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2797.404215] Lustre: Failing over lustre-MDT0000 [ 2797.580712] Lustre: server umount lustre-MDT0000 complete [ 2818.233632] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2818.241215] LDISKFS-fs (dm-0): recovery complete [ 2818.256639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.704956] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8b890df7 [ 2829.373969] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2831.405349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2831.408295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2835.671614] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2836.783638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2842.729110] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 22:59:06 (1763351946) [ 2847.601363] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2848.921415] Lustre: Failing over lustre-MDT0000 [ 2849.068596] Lustre: server umount lustre-MDT0000 complete [ 2867.438118] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2867.439930] LDISKFS-fs (dm-0): recovery complete [ 2867.449515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.707776] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2867.711188] Lustre: Skipped 11 previous similar messages [ 2870.269887] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2872.853414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2872.853449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2875.322948] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2876.425243] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2882.786766] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 22:59:46 (1763351986) [ 2883.560746] Lustre: 93884:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 61808989-3964-40c4-81ec-dd31f403bbd4 at adminstrative request [ 2891.059436] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 22:59:54 (1763351994) [ 2895.398283] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2896.665405] Lustre: Failing over lustre-MDT0000 [ 2896.855580] Lustre: server umount lustre-MDT0000 complete [ 2905.926571] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2905.928390] LDISKFS-fs (dm-0): recovery complete [ 2905.933787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.197879] Lustre: lustre-MDT0000: Aborting client recovery [ 2906.201333] LustreError: 95166:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2906.207669] Lustre: 95199:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2906.212209] Lustre: 95199:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 61808989-3964-40c4-81ec-dd31f403bbd4@ [ 2906.218309] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2906.223333] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2906.237674] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2906.274287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 2906.277666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 2908.789762] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2911.207206] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2925.783864] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 23:00:29 (1763352029) [ 2930.737748] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2932.021171] Lustre: Failing over lustre-MDT0000 [ 2932.190411] Lustre: server umount lustre-MDT0000 complete [ 2936.800878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2936.813125] LustreError: Skipped 4 previous similar messages [ 2940.372176] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2940.374179] LDISKFS-fs (dm-0): recovery complete [ 2940.386832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.720808] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2940.739142] Lustre: lustre-MDT0000: Aborting client recovery [ 2940.741145] LustreError: 97013:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2940.742217] Lustre: 97046:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2940.748467] Lustre: 97046:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2940.752912] Lustre: 97046:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2940.762035] Lustre: 97046:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2940.765461] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2940.772379] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2940.790367] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2940.831889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 2940.835162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1512 to 0x2c0000401:1537) [ 2943.374731] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2946.020067] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2954.372337] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2954.374347] Lustre: Skipped 1 previous similar message [ 2958.671822] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 23:01:02 (1763352062) [ 2963.018438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2963.989789] Lustre: Failing over lustre-MDT0000 [ 2964.124385] Lustre: server umount lustre-MDT0000 complete [ 2972.138460] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2972.139829] LDISKFS-fs (dm-0): recovery complete [ 2972.149769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.338042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.343939] Lustre: Skipped 12 previous similar messages [ 2972.383117] Lustre: lustre-MDT0000: Aborting client recovery [ 2972.385559] LustreError: 98875:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2972.391650] Lustre: 98908:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2972.396228] Lustre: 98908:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2972.402361] Lustre: 98908:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2972.407748] Lustre: 98908:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2972.413847] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2972.422197] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2972.436535] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2972.467957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 2972.469191] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 2974.708219] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2977.762972] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2989.468667] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 23:01:33 (1763352093) [ 2990.057778] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2990.060174] LustreError: 81663:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1de9dbad80 x1849005626757120/t201863462916(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:531/0 lens 512/456 e 0 to 0 dl 1763352106 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2993.008803] Lustre: Failing over lustre-MDT0000 [ 2993.136902] Lustre: server umount lustre-MDT0000 complete [ 2998.062388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.260973] Lustre: lustre-MDT0000: Aborting client recovery [ 2998.262925] LustreError: 100182:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2998.263783] Lustre: 100215:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2998.270572] Lustre: 100215:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2998.275060] Lustre: 100215:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2998.281363] Lustre: 100215:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2998.285393] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2998.294190] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2998.306542] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2998.336326] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1601) [ 2998.337238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 3000.344316] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3003.361972] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3015.896940] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3016.977567] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 23:02:00 (1763352120) [ 3021.443611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3023.055818] Lustre: Failing over lustre-MDT0000 [ 3023.236339] Lustre: server umount lustre-MDT0000 complete [ 3030.540102] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3030.542193] LDISKFS-fs (dm-0): recovery complete [ 3030.552317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.775737] Lustre: lustre-MDT0000: Aborting client recovery [ 3030.778832] LustreError: 102128:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3030.784349] Lustre: 102162:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3030.789913] Lustre: 102162:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3030.794877] Lustre: 102162:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3030.802376] Lustre: 102162:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3030.810194] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3030.817352] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3030.826670] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3030.855659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 3030.863434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633) [ 3033.140462] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3036.130113] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3048.610602] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:02:32 (1763352152) [ 3072.642732] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3073.806736] Lustre: Failing over lustre-MDT0000 [ 3074.055907] Lustre: server umount lustre-MDT0000 complete [ 3091.832209] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3091.834754] LDISKFS-fs (dm-0): recovery complete [ 3091.842578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.352857] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3097.127520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3097.137972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3099.740284] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.752230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3116.262349] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:03:40 (1763352220) [ 3133.213640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3139.640824] Lustre: Failing over lustre-MDT0000 [ 3139.923450] Lustre: server umount lustre-MDT0000 complete [ 3143.136968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.142025] Lustre: Skipped 58 previous similar messages [ 3156.620453] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3156.622692] LDISKFS-fs (dm-0): recovery complete [ 3156.629200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.962120] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3162.084739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3162.087723] Lustre: Skipped 58 previous similar messages [ 3164.266989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3164.267572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3166.116668] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3166.874685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3179.314514] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 23:04:43 (1763352283) [ 3180.267736] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3180.709226] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3180.713124] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3183.386232] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 23:04:47 (1763352287) [ 3197.721675] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3203.431168] Lustre: Failing over lustre-OST0000 [ 3205.491076] LustreError: 108620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3205.493075] LustreError: 108620:0:(obd_class.h:479:obd_check_dev()) Skipped 103 previous similar messages [ 3205.517133] Lustre: server umount lustre-OST0000 complete [ 3222.110534] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3222.113361] LDISKFS-fs (dm-2): recovery complete [ 3222.123363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3223.908665] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3223.913150] Lustre: Skipped 7 previous similar messages [ 3224.119945] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3224.125896] Lustre: Skipped 7 previous similar messages [ 3224.895267] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3273.613795] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 23:06:17 (1763352377) [ 3277.086344] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3278.411538] Lustre: Failing over lustre-MDT0000 [ 3278.587844] Lustre: server umount lustre-MDT0000 complete [ 3294.197178] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3294.199066] LDISKFS-fs (dm-0): recovery complete [ 3294.203984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.275945] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.281985] LustreError: Skipped 12 previous similar messages [ 3296.031737] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3299.842110] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3299.842153] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3301.348666] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3302.127449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3316.192244] Lustre: 111276:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763352405/real 1763352405] req@ffff8a1ded51f480 x1849005643643136/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1763352421 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295 [ 3316.205455] Lustre: 111276:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3316.209750] LustreError: 111276:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3316.212670] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3316.798961] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 23:07:00 (1763352420) [ 3317.280565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3319.278592] LustreError: 9490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3324.384129] LustreError: 9490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3324.389444] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3324.412402] LustreError: 8787:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 3325.338100] LustreError: 9490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3330.528207] LustreError: 9490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3330.533752] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3331.438406] LustreError: 9490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3336.672535] LustreError: 9490:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3336.677765] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3337.556843] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3342.816137] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3343.607492] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3348.960212] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3348.963958] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3348.967575] Lustre: Skipped 1 previous similar message [ 3355.860867] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3355.863664] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3361.248134] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3361.251294] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3367.392193] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3367.397231] Lustre: Skipped 2 previous similar messages [ 3374.288584] LustreError: 6522:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3374.292081] LustreError: 6522:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3379.680146] LustreError: 6522:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3379.684096] LustreError: 6522:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3383.161576] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 23:08:07 (1763352487) [ 3383.923050] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3394.264593] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3399.378260] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3404.498653] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3409.619400] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3409.622327] Lustre: Skipped 1 previous similar message [ 3414.743660] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3423.968457] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3423.972071] Lustre: 6520:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a1de9d91f80 x1849005629399296/t0(0) o38->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1763352509 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3424.980073] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3424.984657] Lustre: Skipped 3 previous similar messages [ 3424.987022] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3449.553825] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3449.558284] Lustre: Skipped 1 previous similar message [ 3465.040178] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3465.044208] Lustre: 6520:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a1ddf54bb80 x1849005629404288/t0(0) o38->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1763352550 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3468.566415] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3494.104465] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3494.113767] Lustre: Skipped 3 previous similar messages [ 3508.616201] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3508.619924] Lustre: 6520:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a1dfbd39c00 x1849005629407872/t0(0) o38->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1763352594 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3509.458470] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3509.462966] Lustre: Skipped 1 previous similar message [ 3509.464983] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3535.059803] Lustre: lustre-MDT0000: Export ffff8a1cc80f1800 already connecting from 192.168.202.31@tcp [ 3535.064172] Lustre: Skipped 2 previous similar messages [ 3549.512091] LustreError: 6520:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3549.515785] Lustre: 6520:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a1ded015500 x1849005629410816/t0(0) o38->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1763352634 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3550.417873] LustreError: 9490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3590.472368] LustreError: 9490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3590.478492] Lustre: 9490:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a1dd17a1180 x1849005629413888/t0(0) o38->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1763352675 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3591.379468] LustreError: 81663:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3597.001097] LustreError: 81663:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 3599.366837] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 23:11:43 (1763352703) [ 3602.964549] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3604.960311] Lustre: Failing over lustre-MDT0000 [ 3605.091567] Lustre: server umount lustre-MDT0000 complete [ 3606.739926] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.747312] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 200 previous similar messages [ 3607.008624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3607.011749] LustreError: Skipped 7 previous similar messages [ 3610.343267] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3610.345085] LDISKFS-fs (dm-0): recovery complete [ 3610.349732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3610.486145] Lustre: *** cfs_fail_loc=712, val=0*** [ 3610.490207] LustreError: 8407:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8a1ded254380 x1849005643788544/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3610.517547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.521318] Lustre: Skipped 6 previous similar messages [ 3610.542903] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3610.543040] Lustre: lustre-MDT0000: Aborting client recovery [ 3610.546578] Lustre: Skipped 19 previous similar messages [ 3610.551401] LustreError: 116010:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3610.555060] Lustre: 116043:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3610.558771] Lustre: 116043:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3610.562496] Lustre: 116043:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3610.567232] Lustre: 116043:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3610.570890] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3610.578905] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3610.587254] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3610.616673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3610.616782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3612.413497] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3615.714377] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3623.689931] Lustre: Failing over lustre-MDT0000 [ 3623.851841] Lustre: server umount lustre-MDT0000 complete [ 3637.747250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3639.549282] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3643.387772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3643.387846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3645.113320] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3645.872253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3650.141313] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 23:12:34 (1763352754) [ 3650.185991] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 3650.189412] Lustre: Skipped 2 previous similar messages [ 3653.893256] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 23:12:38 (1763352758) [ 3654.328498] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3654.330747] LustreError: 6529:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1dffadb800 x1849005629470720/t0(0) o700->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:440/0 lens 264/248 e 0 to 0 dl 1763352770 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3672.458936] Lustre: Failing over lustre-MDT0000 [ 3672.680860] Lustre: server umount lustre-MDT0000 complete [ 3686.701171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.534551] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3692.046708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3692.050042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3693.653735] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3694.397728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3699.144828] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:13:23 (1763352803) [ 3700.194987] Lustre: Failing over lustre-OST0000 [ 3700.265577] Lustre: server umount lustre-OST0000 complete [ 3713.729260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3715.986335] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3719.641898] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3720.323912] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3786.635739] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:14:50 (1763352890) [ 3789.838349] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3790.830060] Lustre: Failing over lustre-MDT0000 [ 3790.996676] Lustre: server umount lustre-MDT0000 complete [ 3794.401371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.407528] Lustre: Skipped 22 previous similar messages [ 3806.068970] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3806.070600] LDISKFS-fs (dm-0): recovery complete [ 3806.074534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.176490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3806.179514] Lustre: Skipped 3 previous similar messages [ 3808.046591] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3811.299430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3811.302576] Lustre: Skipped 25 previous similar messages [ 3811.504211] Lustre: *** cfs_fail_loc=216, val=0*** [ 3811.504499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3811.506823] LustreError: 121818:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3812.576972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 3874.880872] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 23:16:19 (1763352979) [ 3883.946795] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 23:16:28 (1763352988) [ 3885.270193] Lustre: Failing over lustre-MDT0000 [ 3885.360342] LustreError: 122926:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3885.363488] LustreError: 122926:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 3885.403986] Lustre: server umount lustre-MDT0000 complete [ 3898.984806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.049823] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3899.054346] LustreError: Skipped 4 previous similar messages [ 3899.092225] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (not set up) [ 3900.759115] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3900.770490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3900.774356] Lustre: Skipped 5 previous similar messages [ 3904.484249] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3904.486865] LustreError: 123381:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1cc86c9880 x1849005629588992/t0(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:731/0 lens 328/344 e 0 to 0 dl 1763353061 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 3961.042062] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 3961.051090] Lustre: 123381:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3961.062686] Lustre: lustre-MDT0000: Recovery over after 1:01, of 2 clients 2 recovered and 0 were evicted. [ 3961.067234] Lustre: Skipped 5 previous similar messages [ 3961.085556] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 3961.085556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 3962.814060] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3963.538542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3967.904227] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 23:17:52 (1763353072) [ 3969.400230] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3973.012638] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3973.788967] Lustre: Failing over lustre-MDT0000 [ 3973.950288] Lustre: server umount lustre-MDT0000 complete [ 3988.818461] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3988.819666] LDISKFS-fs (dm-0): recovery complete [ 3988.825055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.591218] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3994.110521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 3994.110556] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 3995.560393] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3996.235576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4000.241418] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 23:18:24 (1763353104) [ 4000.736856] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4005.236768] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4006.016770] Lustre: Failing over lustre-MDT0000 [ 4006.121319] Lustre: server umount lustre-MDT0000 complete [ 4021.222630] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4021.224930] LDISKFS-fs (dm-0): recovery complete [ 4021.231100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.930558] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4026.871536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4026.871572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 4028.306775] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4028.949244] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4032.891255] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 23:18:57 (1763353137) [ 4033.358990] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4037.923961] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4038.727316] Lustre: Failing over lustre-MDT0000 [ 4038.826324] Lustre: server umount lustre-MDT0000 complete [ 4053.673061] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4053.674918] LDISKFS-fs (dm-0): recovery complete [ 4053.680200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.527762] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4059.127182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4059.127466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3201) [ 4064.079744] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 23:19:28 (1763353168) [ 4065.514885] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4065.516704] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4065.518504] LustreError: 6524:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1dff8a4380 x1849005629640320/t257698037777(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:96/0 lens 392/456 e 0 to 0 dl 1763353181 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4066.561881] Lustre: Failing over lustre-MDT0000 [ 4066.665741] Lustre: server umount lustre-MDT0000 complete [ 4080.355693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.255267] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4085.750431] Lustre: 6524:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dfbca6a00 x1849005629640320/t257698037777(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:117/0 lens 392/456 e 0 to 0 dl 1763353202 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4085.753973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3233) [ 4085.754169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 4087.128219] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4087.800933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4091.867367] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 23:19:56 (1763353196) [ 4092.302190] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4092.304798] LustreError: 81663:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1cc3a79c00 x1849005629652736/t261993005072(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:167/0 lens 504/448 e 0 to 0 dl 1763353252 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4096.857695] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4097.694637] Lustre: Failing over lustre-MDT0000 [ 4097.810533] Lustre: server umount lustre-MDT0000 complete [ 4113.000475] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4113.002381] LDISKFS-fs (dm-0): recovery complete [ 4113.009444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.729306] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4118.519257] Lustre: 114259:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dfbf46300 x1849005629652736/t261993005072(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:193/0 lens 504/2880 e 0 to 0 dl 1763353278 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4118.520708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3265) [ 4118.520730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 4120.010291] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4120.742659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4124.559728] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 23:20:28 (1763353228) [ 4124.959807] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4124.961069] LustreError: 6521:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1df4c21f80 x1849005629666560/t266287972368(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:200/0 lens 504/448 e 0 to 0 dl 1763353285 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4126.265618] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4128.972610] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4129.648114] Lustre: Failing over lustre-MDT0000 [ 4129.743408] Lustre: server umount lustre-MDT0000 complete [ 4144.636381] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4144.638414] LDISKFS-fs (dm-0): recovery complete [ 4144.647769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.369657] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4150.260304] Lustre: 41999:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dff8a5500 x1849005629667456/t266287972369(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:181/0 lens 392/456 e 0 to 0 dl 1763353266 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4150.263768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3297) [ 4150.263769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4150.271889] Lustre: 41999:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4155.280866] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 23:20:59 (1763353259) [ 4155.790350] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4155.792483] Lustre: Skipped 1 previous similar message [ 4155.794487] LustreError: 6521:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1ded109500 x1849005629679360/t270582939664(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:231/0 lens 504/448 e 0 to 0 dl 1763353316 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4155.805508] LustreError: 6521:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 4157.139348] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4157.141570] Lustre: Skipped 1 previous similar message [ 4160.237980] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4160.942032] Lustre: Failing over lustre-MDT0000 [ 4161.021282] Lustre: server umount lustre-MDT0000 complete [ 4176.222129] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4176.224214] LDISKFS-fs (dm-0): recovery complete [ 4176.228892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.003491] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4181.494991] Lustre: 9490:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dfce91f80 x1849005629679360/t270582939664(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:256/0 lens 504/2880 e 0 to 0 dl 1763353341 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4181.497938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3329) [ 4181.497942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4186.202624] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 23:21:30 (1763353290) [ 4186.640986] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4188.003771] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4188.005678] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4188.007533] LustreError: 6524:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1cc503f480 x1849005629692032/t274877906960(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:219/0 lens 392/456 e 0 to 0 dl 1763353304 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4192.188328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4192.976721] Lustre: Failing over lustre-MDT0000 [ 4193.086793] Lustre: server umount lustre-MDT0000 complete [ 4207.073112] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.079740] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 187 previous similar messages [ 4208.232867] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4208.234849] LDISKFS-fs (dm-0): recovery complete [ 4208.240930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.929709] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4213.747902] Lustre: 6524:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1de9dbd880 x1849005629692032/t274877906960(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:245/0 lens 392/456 e 0 to 0 dl 1763353330 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4213.750953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 4213.750954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 4219.288970] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 23:22:03 (1763353323) [ 4219.722194] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4279.521706] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnecting [ 4279.526234] Lustre: Skipped 4 previous similar messages [ 4279.531537] Lustre: 6521:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1ded5e6300 x1849005629702656/t279172874255(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:354/0 lens 664/3488 e 0 to 0 dl 1763353439 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4282.286727] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 23:23:06 (1763353386) [ 4285.389946] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4286.115912] Lustre: Failing over lustre-MDT0000 [ 4286.215908] Lustre: server umount lustre-MDT0000 complete [ 4290.528780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4290.532118] LustreError: Skipped 11 previous similar messages [ 4301.373256] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4301.375278] LDISKFS-fs (dm-0): recovery complete [ 4301.381363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.522450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.525629] Lustre: Skipped 13 previous similar messages [ 4301.567725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4301.570962] Lustre: Skipped 15 previous similar messages [ 4303.176748] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4306.950715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 4306.951131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361) [ 4308.455912] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4309.113974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4323.498690] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 23:23:47 (1763353427) [ 4327.067304] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4327.808235] Lustre: Failing over lustre-MDT0000 [ 4327.903642] Lustre: server umount lustre-MDT0000 complete [ 4342.906069] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4342.907741] LDISKFS-fs (dm-0): recovery complete [ 4342.916246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.722272] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4348.414345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 4348.414598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 4349.877056] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4350.563790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4353.175090] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4358.776744] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 23:24:23 (1763353463) [ 4375.068503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4375.830530] Lustre: Failing over lustre-MDT0000 [ 4376.086447] Lustre: server umount lustre-MDT0000 complete [ 4390.823415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4390.825333] LDISKFS-fs (dm-0): recovery complete [ 4390.830323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.492121] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4396.109506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4396.109562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4397.682337] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4398.378855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4417.144174] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 23:25:21 (1763353521) [ 4420.670829] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4421.412130] Lustre: Failing over lustre-MDT0000 [ 4421.551324] Lustre: server umount lustre-MDT0000 complete [ 4421.600366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.605186] Lustre: Skipped 53 previous similar messages [ 4436.339416] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4436.341239] LDISKFS-fs (dm-0): recovery complete [ 4436.345369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.166770] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4441.574703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4441.577644] Lustre: Skipped 53 previous similar messages [ 4441.629858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 4441.629952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 4443.092267] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4443.724204] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4446.797394] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 4447.416820] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4449.821381] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 23:25:54 (1763353554) [ 4455.776342] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4516.977418] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4516.979179] Lustre: Skipped 1 previous similar message [ 4516.980879] LustreError: 6520:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1ded109880 x1849005632492672/t296352743435(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:592/0 lens 66040/440 e 0 to 0 dl 1763353677 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4516.987584] LustreError: 6520:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 4576.467268] Lustre: 114259:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dd212dc50 x1849005632492672/t296352743435(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:651/0 lens 66040/440 e 0 to 0 dl 1763353736 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4579.367655] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4579.909206] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 23:28:04 (1763353684) [ 4583.964352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4584.773122] Lustre: Failing over lustre-MDT0000 [ 4584.898214] LustreError: 148297:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4584.900306] LustreError: 148297:0:(obd_class.h:479:obd_check_dev()) Skipped 103 previous similar messages [ 4584.933620] Lustre: server umount lustre-MDT0000 complete [ 4599.213238] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4599.215152] LDISKFS-fs (dm-0): recovery complete [ 4599.220422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.272439] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.276932] LustreError: Skipped 12 previous similar messages [ 4600.692250] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4602.065469] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4602.069678] Lustre: Skipped 12 previous similar messages [ 4604.576377] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4604.579511] Lustre: Skipped 12 previous similar messages [ 4604.595993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 4604.596045] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 4605.813430] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4606.382395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4610.243174] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 23:28:34 (1763353714) [ 4616.719294] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4621.019854] Lustre: Failing over lustre-OST0000 [ 4621.060201] Lustre: server umount lustre-OST0000 complete [ 4635.671321] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4635.672979] LDISKFS-fs (dm-2): recovery complete [ 4635.678105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4637.779241] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4649.360714] Lustre: Failing over lustre-OST0000 [ 4649.410479] Lustre: server umount lustre-OST0000 complete [ 4662.416876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4664.436332] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4667.636577] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4668.222907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4702.500568] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 23:30:06 (1763353806) [ 4703.441907] Lustre: Failing over lustre-MDT0000 [ 4703.690233] Lustre: server umount lustre-MDT0000 complete [ 4716.576201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4718.044712] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4722.163994] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 4722.164024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 4729.592761] Lustre: Failing over lustre-MDT0000 [ 4729.770745] Lustre: server umount lustre-MDT0000 complete [ 4742.754776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.309646] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4748.277755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 4748.277862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 4749.492365] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4750.058177] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4753.636793] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 23:30:57 (1763353857) [ 4765.031103] Lustre: Failing over lustre-OST0000 [ 4765.066539] Lustre: server umount lustre-OST0000 complete [ 4777.864947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4779.731871] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4782.872203] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4783.432969] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4787.167188] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 23:31:31 (1763353891) [ 4787.830569] Lustre: Failing over lustre-MDT0000 [ 4788.076872] Lustre: server umount lustre-MDT0000 complete [ 4791.263825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.298729] Lustre: *** cfs_fail_loc=605, val=0*** [ 4791.300156] LustreError: 157133:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0a0cce0 failed: rc = -95 [ 4791.303850] LustreError: 157133:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 4791.306866] LustreError: 157133:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 4791.308866] LustreError: 157133:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 4791.311327] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4791.313126] LustreError: 157133:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 4791.319209] Lustre: server umount lustre-MDT0000 complete [ 4791.321058] LustreError: 157133:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4793.990910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.375843] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4798.109719] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 23:31:42 (1763353902) [ 4799.474097] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 4799.474102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 4802.048050] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4803.090754] Lustre: Failing over lustre-MDT0000 [ 4803.271411] Lustre: server umount lustre-MDT0000 complete [ 4809.696610] LustreError: 77171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.701706] LustreError: 77171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 152 previous similar messages [ 4817.301977] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4817.303191] LDISKFS-fs (dm-0): recovery complete [ 4817.305438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.731628] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4822.226800] Lustre: *** cfs_fail_loc=707, val=0*** [ 4881.635263] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 4881.646991] Lustre: 159287:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4881.650443] Lustre: 159287:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 4881.755607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 4881.755618] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 4883.193170] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4883.781895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4887.832562] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 23:33:12 (1763353992) [ 4911.426620] LustreError: 114259:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1de9c36300 x1849005633372672/t0(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:187/0 lens 664/0 e 0 to 0 dl 1763354027 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4911.436021] LustreError: 114259:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 4922.464113] LustreError: 114259:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4922.469766] LustreError: 41999:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1df17a1880 x1849005633373952/t0(0) o35->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:198/0 lens 392/0 e 0 to 0 dl 1763354038 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4927.970034] LustreError: 8787:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1dee66d850 x1849005645495424/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:204/0 lens 544/0 e 0 to 0 dl 1763354044 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 4927.979425] LustreError: 8787:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 52 previous similar messages [ 4934.195156] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 23:33:58 (1763354038) [ 4957.501815] LustreError: 8411:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 4968.528106] LustreError: 8411:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 4971.595453] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 23:34:35 (1763354075) [ 4994.698569] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1cc5592d80 x1849005633397760/t0(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:271/0 lens 576/0 e 0 to 0 dl 1763354111 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4994.705071] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4999.800137] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4999.805171] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1df4c58380 x1849005633398656/t0(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:276/0 lens 664/0 e 0 to 0 dl 1763354116 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5010.592057] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5010.597284] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1df16cdf80 x1849005633420416/t0(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:327/0 lens 664/0 e 0 to 0 dl 1763354167 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5010.608171] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5022.678323] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 23:35:26 (1763354126) [ 5107.825223] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 23:36:52 (1763354212) [ 5130.833104] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1ded266300 x1849005633477632/t0(0) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:407/0 lens 576/0 e 0 to 0 dl 1763354247 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5130.841583] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 5130.844960] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5130.848297] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5131.264101] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5147.090407] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5147.093479] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5147.512098] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5147.515978] LustreError: 6520:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5162.922072] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a1df17f9c00 x1849005633496832/t0(0) o36->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:439/0 lens 504/0 e 0 to 0 dl 1763354279 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5162.930182] LustreError: 6520:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [ 5174.906982] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 23:37:59 (1763354279) [ 5199.356930] Lustre: DEBUG MARKER: phase 2 [ 5202.411761] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 23:38:26 (1763354306) [ 5273.107821] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 23:39:37 (1763354377) [ 5273.568691] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5274.112070] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 23:39:38 (1763354378) [ 5275.921417] Lustre: DEBUG MARKER: Started rundbench load pid=133766 ... [ 5279.698278] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5281.257420] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5281.934646] Lustre: Failing over lustre-MDT0000 [ 5282.147674] LustreError: 166179:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5282.150652] LustreError: 166179:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 5282.187151] Lustre: server umount lustre-MDT0000 complete [ 5283.297063] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.301813] Lustre: Skipped 32 previous similar messages [ 5296.719940] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5296.722126] LDISKFS-fs (dm-0): recovery complete [ 5296.725565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.767278] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5296.771480] LustreError: Skipped 4 previous similar messages [ 5296.846730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.849386] Lustre: Skipped 11 previous similar messages [ 5296.864978] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5296.867119] Lustre: Skipped 11 previous similar messages [ 5298.286071] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5298.384711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5298.388227] Lustre: Skipped 7 previous similar messages [ 5302.242502] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5302.247889] Lustre: Skipped 29 previous similar messages [ 5302.258754] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5302.262370] Lustre: Skipped 7 previous similar messages [ 5302.279709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 5302.279727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 5303.486321] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5304.080885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5309.062430] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5310.601885] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5311.258238] Lustre: Failing over lustre-MDT0001 [ 5311.268271] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 5311.477566] Lustre: server umount lustre-MDT0001 complete [ 5312.481230] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5312.484052] LustreError: Skipped 9 previous similar messages [ 5325.640383] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5325.642624] LDISKFS-fs (dm-1): recovery complete [ 5325.647603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5327.003203] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5331.175142] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:782 to 0x2c0000400:801) [ 5331.175182] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:801 to 0x280000400:833) [ 5332.379356] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5333.000223] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5337.878985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5339.411639] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5340.083864] Lustre: Failing over lustre-MDT0000 [ 5340.310658] Lustre: server umount lustre-MDT0000 complete [ 5354.709459] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5354.711294] LDISKFS-fs (dm-0): recovery complete [ 5354.716364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.329098] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5360.124057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 5360.124060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 5361.442320] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5362.105123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5367.246182] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5368.809225] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 5369.492467] Lustre: Failing over lustre-MDT0001 [ 5369.498950] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 5369.530021] LustreError: 6521:0:(ldlm_lockd.c:1452:ldlm_handle_enqueue()) ### lock on destroyed export 00000000b21b3701 ns: mdt-lustre-MDT0001_UUID lock: ffff8a1ccbac8200/0x7ca51b7b8b9bd5c3 lrc: 3/0,0 mode: CW/CW res: [0x2400007ed:0x346:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.202.31@tcp remote: 0x79ef1da4e1417118 expref: 4 pid: 6521 timeout: 0 lvb_type: 0 [ 5373.137204] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 5373.139832] Lustre: Skipped 3 previous similar messages [ 5375.725419] Lustre: server umount lustre-MDT0001 complete [ 5389.901980] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5389.903777] LDISKFS-fs (dm-1): recovery complete [ 5389.907681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5391.368528] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5395.813620] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1227 to 0x280000400:1249) [ 5395.813622] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1196 to 0x2c0000400:1217) [ 5396.989347] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5397.544284] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5421.718135] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 23:42:05 (1763354525) [ 5544.890881] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5555.677765] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 5556.343201] Lustre: Failing over lustre-MDT0001 [ 5556.526456] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5556.528270] Lustre: Skipped 3 previous similar messages [ 5561.945714] Lustre: server umount lustre-MDT0001 complete [ 5564.384685] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.388761] LustreError: 9490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 60 previous similar messages [ 5576.142619] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5576.143855] LDISKFS-fs (dm-1): recovery complete [ 5576.147785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5577.678234] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5583.434535] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2728 to 0x2c0000400:2753) [ 5583.456608] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2760 to 0x280000400:2785) [ 5584.664674] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5585.250577] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5709.528637] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5720.352306] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 5721.051883] Lustre: Failing over lustre-MDT0001 [ 5721.069831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5721.073120] Lustre: Skipped 5 previous similar messages [ 5727.311992] Lustre: server umount lustre-MDT0001 complete [ 5741.438842] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5741.440620] LDISKFS-fs (dm-1): recovery complete [ 5741.444064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5742.807853] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5748.660893] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4071 to 0x280000400:4097) [ 5748.676917] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4039 to 0x2c0000400:4065) [ 5749.845322] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5750.390227] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5774.430465] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 23:47:58 (1763354878) [ 5897.315576] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5908.038369] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 5908.703071] Lustre: Failing over lustre-MDT0001 [ 5908.718712] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.31@tcp (stopping) [ 5908.721660] Lustre: Skipped 5 previous similar messages [ 5908.995950] LustreError: 176798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5908.997860] LustreError: 176798:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 5909.049173] Lustre: server umount lustre-MDT0001 complete [ 5910.497073] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.500975] Lustre: Skipped 18 previous similar messages [ 5923.283478] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5923.285786] LDISKFS-fs (dm-1): recovery complete [ 5923.290636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5923.395763] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5923.405363] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5923.407200] Lustre: Skipped 5 previous similar messages [ 5924.571647] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5924.573542] Lustre: Skipped 5 previous similar messages [ 5924.736443] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5928.418336] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5928.420125] Lustre: Skipped 19 previous similar messages [ 5929.381537] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5929.384819] Lustre: Skipped 5 previous similar messages [ 5929.407606] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4640 to 0x280000400:4673) [ 5929.408205] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4608 to 0x2c0000400:4641) [ 5930.666215] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5931.322638] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6055.557614] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6066.310342] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 6066.997728] Lustre: Failing over lustre-MDT0000 [ 6067.024191] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 6071.777343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6071.779858] LustreError: Skipped 4 previous similar messages [ 6072.928426] Lustre: server umount lustre-MDT0000 complete [ 6087.133316] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6087.135038] LDISKFS-fs (dm-0): recovery complete [ 6087.139488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.178769] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6087.184212] LustreError: Skipped 1 previous similar message [ 6087.252540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6087.256151] Lustre: Skipped 5 previous similar messages [ 6088.590555] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6093.396959] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8722 to 0x2c0000401:8737) [ 6093.397031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8753 to 0x280000401:8769) [ 6094.651181] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6095.244904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6098.755710] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 23:53:23 (1763355203) [ 6222.020783] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6232.827937] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 6233.516608] Lustre: Failing over lustre-MDT0000 [ 6233.592895] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 6233.594668] Lustre: Skipped 5 previous similar messages [ 6233.804240] Lustre: server umount lustre-MDT0000 complete [ 6235.617099] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.621516] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 50 previous similar messages [ 6247.843888] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6247.845685] LDISKFS-fs (dm-0): recovery complete [ 6247.849174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.205245] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6256.139973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10672 to 0x2c0000401:10689) [ 6256.147665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10704 to 0x280000401:10721) [ 6257.502663] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6258.183794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6382.412424] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6393.171795] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 6393.812755] Lustre: Failing over lustre-MDT0001 [ 6393.841603] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6394.093129] Lustre: server umount lustre-MDT0001 complete [ 6408.185602] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6408.187645] LDISKFS-fs (dm-1): recovery complete [ 6408.192609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6409.682331] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6413.817903] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5087 to 0x280000400:5121) [ 6413.817955] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5054 to 0x2c0000400:5089) [ 6415.087543] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6415.725309] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6419.735061] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 23:58:44 (1763355524) [ 6425.755626] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6427.243079] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6427.882865] Lustre: Failing over lustre-OST0000 [ 6427.926989] Lustre: server umount lustre-OST0000 complete [ 6442.068461] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6442.069687] LDISKFS-fs (dm-2): recovery complete [ 6442.072188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6443.957042] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6447.297564] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6447.892647] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6456.804526] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6458.294703] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6458.921510] Lustre: Failing over lustre-OST0000 [ 6458.962929] Lustre: server umount lustre-OST0000 complete [ 6473.110475] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6473.111632] LDISKFS-fs (dm-2): recovery complete [ 6473.114942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6475.013496] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6478.332954] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6479.044370] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6484.688572] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 23:59:48 (1763355588) [ 6607.782787] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6610.342110] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6621.142394] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 6621.833356] Lustre: Failing over lustre-MDT0000 [ 6621.965509] LustreError: 188207:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6621.967487] LustreError: 188207:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 6622.015479] Lustre: server umount lustre-MDT0000 complete [ 6623.200506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.205115] Lustre: Skipped 15 previous similar messages [ 6623.277904] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763355728 with bad export cookie 8981615249303234686 [ 6623.279707] Lustre: Failing over lustre-MDT0001 [ 6623.280816] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6623.562039] Lustre: server umount lustre-MDT0001 complete [ 6638.633365] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6638.635850] LDISKFS-fs (dm-1): recovery complete [ 6638.640251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6638.666783] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6638.669138] LDISKFS-fs (dm-0): recovery complete [ 6638.672231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.727311] LustreError: 189505:0:(llog.c:1616:llog_backup()) MGC192.168.202.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6638.729836] Lustre: 189505:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6642.656190] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763355732/real 1763355732] req@ffff8a1cccf36680 x1849005687771392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763355748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6642.656190] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763355732/real 1763355732] req@ffff8a1cccf34e00 x1849005687771264/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763355748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6642.656208] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6642.665693] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6647.777163] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1ccd4c1880 x1849005687774976/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6647.785437] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 13 previous similar messages [ 6647.910792] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6647.913613] Lustre: Skipped 5 previous similar messages [ 6648.005342] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6648.006731] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6648.007871] Lustre: Skipped 5 previous similar messages [ 6648.010609] Lustre: Skipped 17 previous similar messages [ 6649.404324] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6649.479153] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6654.055543] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 6654.058529] Lustre: Skipped 5 previous similar messages [ 6654.073878] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5054 to 0x2c0000400:5121) [ 6654.073918] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5087 to 0x280000400:5153) [ 6658.129278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12761 to 0x2c0000401:12801) [ 6658.129561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12792 to 0x280000401:12833) [ 6659.343987] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6659.928980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6660.476097] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6784.692236] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6787.451500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6798.165989] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 6798.813304] Lustre: Failing over lustre-MDT0001 [ 6798.815740] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 6798.817523] LustreError: Skipped 6 previous similar messages [ 6798.819039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6798.820891] Lustre: Skipped 1 previous similar message [ 6798.984733] Lustre: server umount lustre-MDT0001 complete [ 6799.840136] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763355743/real 1763355743] req@ffff8a1ccbea5f80 x1849005687772928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763355759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6799.847247] Lustre: 3658:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 6800.229701] Lustre: Failing over lustre-MDT0000 [ 6800.450811] Lustre: server umount lustre-MDT0000 complete [ 6815.397178] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6815.400557] LDISKFS-fs (dm-1): recovery complete [ 6815.404979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6815.464996] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6815.466164] LDISKFS-fs (dm-0): recovery complete [ 6815.468660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.501901] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.504682] LustreError: Skipped 2 previous similar messages [ 6815.507483] LustreError: 192746:0:(llog.c:1616:llog_backup()) MGC192.168.202.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6815.507752] LustreError: 192748:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.202.131@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8a1dc7eae400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6815.511879] Lustre: 192746:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6815.591851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6815.594075] Lustre: Skipped 6 previous similar messages [ 6816.970119] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6817.115738] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6821.750502] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5054 to 0x2c0000400:5153) [ 6821.750527] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5087 to 0x280000400:5185) [ 6826.802192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12792 to 0x280000401:12865) [ 6826.805495] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12761 to 0x2c0000401:12833) [ 6828.028336] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6828.674702] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6829.267548] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6832.735957] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 00:05:37 (1763355937) [ 6835.438918] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6836.467110] Lustre: Failing over lustre-MDT0000 [ 6836.623268] Lustre: server umount lustre-MDT0000 complete [ 6837.217551] LustreError: 192755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.222354] LustreError: 192755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 63 previous similar messages [ 6850.535083] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6850.536404] LDISKFS-fs (dm-0): recovery complete [ 6850.540040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.884310] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6854.866662] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6871.256469] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [ 6871.268570] Lustre: 193512:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1ccdfda300 x1849005722817792/t335007470281(335007470281) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:637/0 lens 520/3488 e 0 to 0 dl 1763355987 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6871.298314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12867 to 0x280000401:12897) [ 6871.298316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12761 to 0x2c0000401:12865) [ 6872.607744] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6873.103531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6876.558692] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 00:06:20 (1763355980) [ 6879.003672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6879.901724] Lustre: Failing over lustre-MDT0000 [ 6880.044128] Lustre: server umount lustre-MDT0000 complete [ 6894.022221] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6894.023760] LDISKFS-fs (dm-0): recovery complete [ 6894.026388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.418238] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6895.834046] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6895.835441] LustreError: 192757:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1ddf439500 x1849005722817792/t335007470281(335007470281) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:662/0 lens 520/664 e 0 to 0 dl 1763356012 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6912.215665] Lustre: lustre-MDT0000: Client 61808989-3964-40c4-81ec-dd31f403bbd4 (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [ 6912.220028] Lustre: 192755:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1cc86b2300 x1849005722817792/t335007470281(335007470281) o101->61808989-3964-40c4-81ec-dd31f403bbd4@192.168.202.31@tcp:678/0 lens 520/3488 e 0 to 0 dl 1763356028 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6912.249858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12867 to 0x280000401:12929) [ 6912.249881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:12897) [ 6913.627719] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6914.133140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6918.143989] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 00:07:02 (1763356022) [ 6919.603302] Lustre: Failing over lustre-OST0000 [ 6919.645691] Lustre: server umount lustre-OST0000 complete [ 6920.839190] Lustre: Failing over lustre-MDT0000 [ 6920.927875] Lustre: server umount lustre-MDT0000 complete [ 6933.558438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.946033] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6935.621322] Lustre: lustre-MDT0000: Denying connection for new client b13b8b2d-f0d9-4ba6-8f92-219f05218947 (at 192.168.202.31@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6939.126230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:12929) [ 6942.787872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6944.630815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12867 to 0x280000401:12961) [ 6944.664450] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6949.368111] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 00:07:33 (1763356053) [ 6949.718697] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6949.720311] LustreError: 192763:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1ccbceca80 x1849005700172800/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:716/0 lens 1312/4320 e 0 to 0 dl 1763356066 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6952.450539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6953.127670] Lustre: Failing over lustre-MDT0000 [ 6953.463369] Lustre: server umount lustre-MDT0000 complete [ 6966.241084] Lustre: 192783:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763356055/real 1763356055] req@ffff8a1ccbced500 x1849005700172800/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1763356071 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6966.248757] Lustre: 192783:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6967.489109] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6967.490320] LDISKFS-fs (dm-0): recovery complete [ 6967.492953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6968.918802] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6972.923028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:12961) [ 6972.923040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:12993) [ 6974.028687] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6974.535518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6977.972977] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 00:08:02 (1763356082) [ 6978.319098] LustreError: 201240:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1cc7f1e300 x1849005700199168/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:744/0 lens 2520/4320 e 0 to 0 dl 1763356094 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6980.658545] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6983.038581] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6983.613065] Lustre: Failing over lustre-MDT0001 [ 6988.867195] Lustre: server umount lustre-MDT0001 complete [ 7003.356484] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7003.358367] LDISKFS-fs (dm-1): recovery complete [ 7003.363893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7004.754952] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7008.753739] Lustre: 192785:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1ddf53dc00 x1849005736682112/t38654708459(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:42/0 lens 560/2880 e 0 to 0 dl 1763356147 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7008.762351] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5196 to 0x280000400:5217) [ 7008.762382] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5185) [ 7009.866431] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7010.358370] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7014.623912] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 00:08:38 (1763356118) [ 7014.978233] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7014.979539] Lustre: Skipped 1 previous similar message [ 7014.980645] LustreError: 193509:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1df55e5c00 x1849005700232064/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:26/0 lens 2520/4320 e 0 to 0 dl 1763356131 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7017.422331] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7019.911179] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7020.510105] Lustre: Failing over lustre-MDT0000 [ 7020.819149] Lustre: server umount lustre-MDT0000 complete [ 7034.632777] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7034.633916] LDISKFS-fs (dm-0): recovery complete [ 7034.636252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.977708] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7040.020509] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:12993) [ 7040.020575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13025) [ 7041.120320] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7041.668575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7043.578314] Lustre: Failing over lustre-MDT0001 [ 7043.678769] Lustre: server umount lustre-MDT0001 complete [ 7057.655369] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7057.657746] LDISKFS-fs (dm-1): recovery complete [ 7057.661626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7059.034504] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7063.028915] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5228 to 0x280000400:5249) [ 7063.028927] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5196 to 0x2c0000400:5217) [ 7064.202383] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7064.761834] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7068.259212] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 00:09:32 (1763356172) [ 7073.953477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7076.251982] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7076.835576] Lustre: Failing over lustre-MDT0000 [ 7077.062147] Lustre: server umount lustre-MDT0000 complete [ 7078.313853] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356183 with bad export cookie 8981615249306237875 [ 7078.315331] Lustre: Failing over lustre-MDT0001 [ 7078.317382] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7084.001199] LustreError: 208429:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a1ccbf2f800 x1849005700291712/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7084.006040] LustreError: 208429:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7084.103259] Lustre: server umount lustre-MDT0001 complete [ 7098.735848] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7098.737424] LDISKFS-fs (dm-0): recovery complete [ 7098.740247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.741190] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7098.747676] LDISKFS-fs (dm-1): recovery complete [ 7098.751851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7102.944461] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1df4ae9c00 x1849005700292480/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7104.504768] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7104.613323] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7109.367470] Lustre: 210288:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1dfbfc6a00 x1849005736749312/t47244640324(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:144/0 lens 560/2880 e 0 to 0 dl 1763356249 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7109.371890] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5260 to 0x280000400:5281) [ 7109.372146] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5228 to 0x2c0000400:5249) [ 7121.136662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13025) [ 7121.136668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13057) [ 7122.429676] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7122.922271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7123.359090] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7126.716868] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 00:10:31 (1763356231) [ 7127.024764] LustreError: 209547:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1de9c44e00 x1849005736782208/t51539607620(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:138/0 lens 560/448 e 0 to 0 dl 1763356243 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7127.030468] LustreError: 209547:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 7132.360994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7132.984620] Lustre: Failing over lustre-MDT0000 [ 7133.294648] Lustre: server umount lustre-MDT0000 complete [ 7142.618233] Lustre: lustre-MDT0001: Client b13b8b2d-f0d9-4ba6-8f92-219f05218947 (at 192.168.202.31@tcp) reconnecting [ 7142.622213] Lustre: Skipped 2 previous similar messages [ 7142.625633] Lustre: 210300:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1cc843d880 x1849005736782208/t51539607620(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:154/0 lens 560/2880 e 0 to 0 dl 1763356259 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7147.236842] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7147.237946] LDISKFS-fs (dm-0): recovery complete [ 7147.240472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.713319] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7152.637622] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13057) [ 7152.637647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13089) [ 7153.757619] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7154.339189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7157.787146] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 00:11:02 (1763356262) [ 7158.142074] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7158.143537] Lustre: Skipped 2 previous similar messages [ 7160.602390] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7161.212763] Lustre: Failing over lustre-MDT0001 [ 7161.309323] Lustre: server umount lustre-MDT0001 complete [ 7175.525862] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7175.527891] LDISKFS-fs (dm-1): recovery complete [ 7175.533043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7176.914203] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7180.791583] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5270 to 0x2c0000400:5313) [ 7180.791645] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5302 to 0x280000400:5345) [ 7181.862306] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7182.411524] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7186.008050] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 00:11:30 (1763356290) [ 7191.738516] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7194.377548] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7195.112632] Lustre: Failing over lustre-MDT0000 [ 7195.458173] Lustre: server umount lustre-MDT0000 complete [ 7209.293516] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7209.294657] LDISKFS-fs (dm-0): recovery complete [ 7209.297077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.645664] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7214.614060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13121) [ 7214.614108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13089) [ 7215.783873] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7216.299621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7218.205535] Lustre: Failing over lustre-MDT0001 [ 7218.294407] Lustre: server umount lustre-MDT0001 complete [ 7232.358916] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7232.360932] LDISKFS-fs (dm-1): recovery complete [ 7232.364997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7233.717715] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7237.637174] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5356 to 0x280000400:5377) [ 7237.637305] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5345) [ 7238.759539] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7239.268371] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7242.583728] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 00:12:26 (1763356346) [ 7248.276296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7250.671508] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7251.278800] Lustre: Failing over lustre-MDT0000 [ 7251.622985] LustreError: 218828:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 7251.624809] LustreError: 218828:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 7251.650650] Lustre: server umount lustre-MDT0000 complete [ 7252.801597] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356358 with bad export cookie 8981615249306251504 [ 7252.802861] Lustre: Failing over lustre-MDT0001 [ 7252.805076] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7252.908136] Lustre: server umount lustre-MDT0001 complete [ 7267.806281] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7267.809208] LDISKFS-fs (dm-1): recovery complete [ 7267.815524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7267.819144] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7267.820822] LDISKFS-fs (dm-0): recovery complete [ 7267.824192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.904126] LustreError: 220125:0:(llog.c:1616:llog_backup()) MGC192.168.202.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7267.906452] Lustre: 220125:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7268.320763] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7268.324597] Lustre: Skipped 57 previous similar messages [ 7269.344142] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763356358/real 1763356358] req@ffff8a1cc52f5c00 x1849005700445440/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763356374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7269.350908] Lustre: 3657:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7278.560600] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1cc52f5180 x1849005700449280/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7278.649508] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7278.651331] Lustre: Skipped 17 previous similar messages [ 7278.801961] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7278.806078] Lustre: Skipped 18 previous similar messages [ 7280.108451] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7280.255937] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7284.195148] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7284.198732] Lustre: Skipped 58 previous similar messages [ 7284.776723] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 7284.779853] Lustre: Skipped 17 previous similar messages [ 7284.785738] Lustre: 220160:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1df5632a00 x1849005736876928/t60129542212(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:296/0 lens 560/2880 e 0 to 0 dl 1763356401 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7284.792020] Lustre: 220160:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 7284.844961] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5388 to 0x280000400:5409) [ 7284.844961] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5356 to 0x2c0000400:5377) [ 7284.852437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13153) [ 7284.852647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13121) [ 7285.970899] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.480880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7287.003401] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7290.401499] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 00:13:14 (1763356394) [ 7290.778332] LustreError: 8417:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1df58ac700 x1849005700475648/t382252089358(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:302/0 lens 1744/4320 e 0 to 0 dl 1763356407 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7290.784126] LustreError: 8417:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 3 previous similar messages [ 7293.172596] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7293.794426] Lustre: Failing over lustre-MDT0001 [ 7300.154884] Lustre: server umount lustre-MDT0001 complete [ 7314.027088] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7314.028249] LDISKFS-fs (dm-1): recovery complete [ 7314.032129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7315.403396] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7319.543918] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5441) [ 7319.543946] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5409) [ 7320.690758] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7321.241985] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7324.734808] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 00:13:49 (1763356429) [ 7327.517104] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7328.136663] Lustre: Failing over lustre-MDT0000 [ 7328.479481] Lustre: server umount lustre-MDT0000 complete [ 7342.260728] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7342.261914] LDISKFS-fs (dm-0): recovery complete [ 7342.264430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.477804] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7347.704523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13153) [ 7347.704535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13185) [ 7348.784678] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7349.275398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7352.554699] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 00:14:16 (1763356456) [ 7355.335434] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7357.683288] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7358.292383] Lustre: Failing over lustre-MDT0000 [ 7358.650292] Lustre: server umount lustre-MDT0000 complete [ 7372.296293] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7372.298100] LDISKFS-fs (dm-0): recovery complete [ 7372.301791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.662434] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7377.919232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13185) [ 7377.919233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13217) [ 7379.043808] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7379.561075] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7381.445729] Lustre: Failing over lustre-MDT0001 [ 7381.538775] Lustre: server umount lustre-MDT0001 complete [ 7395.351659] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7395.352981] LDISKFS-fs (dm-1): recovery complete [ 7395.355951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7396.628713] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7400.947982] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5441) [ 7400.948018] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5473) [ 7402.078366] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7402.638901] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7406.097747] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 00:15:10 (1763356510) [ 7408.795484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7411.110264] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7411.704273] Lustre: Failing over lustre-MDT0000 [ 7411.922739] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.31@tcp (stopping) [ 7411.924452] Lustre: Skipped 26 previous similar messages [ 7411.980565] Lustre: server umount lustre-MDT0000 complete [ 7413.115592] LustreError: 145205:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356518 with bad export cookie 8981615249306261878 [ 7413.116794] Lustre: Failing over lustre-MDT0001 [ 7413.118820] LustreError: 145205:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7413.122769] LustreError: 229607:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff8a1dea34f000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7418.849290] LustreError: 229607:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a1ccc242a00 x1849005700582528/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7418.854299] LustreError: 229607:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7418.968683] Lustre: server umount lustre-MDT0001 complete [ 7433.723657] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7433.725461] LDISKFS-fs (dm-1): recovery complete [ 7433.729385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7433.733283] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7433.734871] LDISKFS-fs (dm-0): recovery complete [ 7433.738157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.812167] LustreError: 230706:0:(llog.c:1616:llog_backup()) MGC192.168.202.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7433.814726] Lustre: 230706:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7437.842992] LustreError: 8417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.848180] LustreError: 8417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 213 previous similar messages [ 7437.862620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7437.864283] Lustre: Skipped 21 previous similar messages [ 7437.928877] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 7437.931215] LustreError: Skipped 14 previous similar messages [ 7439.206031] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7439.259769] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7444.021973] Lustre: 230717:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a1ded265880 x1849005736951936/t73014444034(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:481/0 lens 496/2888 e 0 to 0 dl 1763356586 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7444.027762] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5505) [ 7444.027805] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5473) [ 7444.036451] Lustre: 230717:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7444.069204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13249) [ 7444.069254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13217) [ 7445.109734] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7445.581124] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7446.090763] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7449.205156] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 00:15:53 (1763356553) [ 7449.555281] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7449.557628] Lustre: Skipped 6 previous similar messages [ 7452.099522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7452.645924] Lustre: Failing over lustre-MDT0000 [ 7452.778390] Lustre: server umount lustre-MDT0000 complete [ 7466.698722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7466.700810] LDISKFS-fs (dm-0): recovery complete [ 7466.704492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.744747] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7466.747267] LustreError: Skipped 12 previous similar messages [ 7468.055080] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7472.129505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13281) [ 7472.129512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13249) [ 7473.190398] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7473.714872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7505.105100] Lustre: lustre-MDT0001: Client b13b8b2d-f0d9-4ba6-8f92-219f05218947 (at 192.168.202.31@tcp) reconnecting [ 7505.108904] Lustre: Skipped 1 previous similar message [ 7507.144785] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 00:16:51 (1763356611) [ 7509.717450] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7510.302207] Lustre: Failing over lustre-MDT0001 [ 7510.387581] Lustre: server umount lustre-MDT0001 complete [ 7524.268085] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7524.269107] LDISKFS-fs (dm-1): recovery complete [ 7524.272352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7525.575158] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7529.459543] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5537) [ 7529.459580] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5505) [ 7530.478519] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7530.941950] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7533.996921] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 00:17:18 (1763356638) [ 7536.684690] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7539.007589] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7539.603356] Lustre: Failing over lustre-MDT0000 [ 7539.788096] Lustre: server umount lustre-MDT0000 complete [ 7553.451494] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7553.452603] LDISKFS-fs (dm-0): recovery complete [ 7553.454700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.732659] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7558.652338] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13281) [ 7558.652382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13313) [ 7559.618546] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7560.060528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7561.865246] Lustre: Failing over lustre-MDT0001 [ 7561.947206] Lustre: server umount lustre-MDT0001 complete [ 7575.586141] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7575.587707] LDISKFS-fs (dm-1): recovery complete [ 7575.590794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7576.870927] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7581.172514] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5569) [ 7581.172720] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5537) [ 7582.200791] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7582.693608] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7585.855973] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 00:18:10 (1763356690) [ 7586.177713] LustreError: 230717:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8a1de9f4b480 x1849005737008640/t85899345922(0) o36->b13b8b2d-f0d9-4ba6-8f92-219f05218947@192.168.202.31@tcp:597/0 lens 496/456 e 0 to 0 dl 1763356702 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7586.183231] LustreError: 230717:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 6 previous similar messages [ 7588.486380] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7590.845521] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7591.450988] Lustre: Failing over lustre-MDT0000 [ 7591.618202] Lustre: server umount lustre-MDT0000 complete [ 7592.754903] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356698 with bad export cookie 8981615249306269375 [ 7592.756160] Lustre: Failing over lustre-MDT0001 [ 7592.759681] LustreError: 6505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7592.870868] Lustre: server umount lustre-MDT0001 complete [ 7607.694893] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7607.695655] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7607.696357] LDISKFS-fs (dm-0): recovery complete [ 7607.697500] LDISKFS-fs (dm-1): recovery complete [ 7607.700027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7607.702089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.781950] LustreError: 241350:0:(llog.c:1616:llog_backup()) MGC192.168.202.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7607.784336] Lustre: 241350:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7618.528834] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a1cc1f66300 x1849005700727296/t0(0) o250->MGC192.168.202.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7618.538929] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 7620.066252] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7620.186508] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7624.826300] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5601) [ 7624.826471] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5569) [ 7624.858096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12867 to 0x2c0000401:13313) [ 7624.858211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13345) [ 7626.086385] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7626.629285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7627.165793] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7630.448440] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 00:18:54 (1763356734) [ 7631.047804] Lustre: 242585:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b13b8b2d-f0d9-4ba6-8f92-219f05218947 at adminstrative request [ 7635.724760] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 00:19:00 (1763356740) [ 7637.472268] Lustre: Failing over lustre-MDT0000 [ 7637.688129] Lustre: server umount lustre-MDT0000 complete [ 7650.263685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.603726] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7655.433557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13409) [ 7655.433627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13397 to 0x280000401:13441) [ 7656.445432] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7656.976858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7660.381836] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 00:19:24 (1763356764) [ 7663.760160] Lustre: Failing over lustre-OST0000 [ 7663.827580] Lustre: server umount lustre-OST0000 complete [ 7676.418503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7678.272365] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7681.312381] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7681.864701] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7685.235994] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 00:19:49 (1763356789) [ 7686.489459] Lustre: Failing over lustre-MDT0000 [ 7686.680495] Lustre: server umount lustre-MDT0000 complete [ 7688.673901] LustreError: 6506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356794 with bad export cookie 8981615249306281905 [ 7688.952535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7690.177687] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7690.848841] Lustre: lustre-MDT0000: Denying connection for new client 00d1f1e3-e590-4af2-bbc0-bbe6e1670853 (at 192.168.202.31@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7690.855774] Lustre: Skipped 2 previous similar messages [ 7694.350104] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13441) [ 7694.350112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13397 to 0x280000401:13473) [ 7698.191574] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 00:20:02 (1763356802) [ 7700.780861] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7701.606043] Lustre: Failing over lustre-OST0000 [ 7701.638383] Lustre: server umount lustre-OST0000 complete [ 7715.434296] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7715.436467] LDISKFS-fs (dm-2): recovery complete [ 7715.440519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7717.339450] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7720.333294] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7720.864736] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7724.078851] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 00:20:28 (1763356828) [ 7726.630844] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7728.402855] Lustre: Failing over lustre-OST0000 [ 7728.435477] Lustre: server umount lustre-OST0000 complete [ 7742.183891] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7742.184983] LDISKFS-fs (dm-2): recovery complete [ 7742.188159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7743.788780] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.31@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:13475 extent [0-4194303]: client csum 432a12b4, server csum 6ac21239 [ 7744.077504] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7747.048275] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7747.578068] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7750.979752] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 00:20:55 (1763356855) [ 7753.297341] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7755.644685] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7757.445472] Lustre: Failing over lustre-MDT0000 [ 7757.751299] Lustre: server umount lustre-MDT0000 complete [ 7758.893602] LustreError: 7448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763356864 with bad export cookie 8981615249306296136 [ 7758.895476] Lustre: Failing over lustre-OST0000 [ 7758.896348] LustreError: 7448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7758.931109] Lustre: server umount lustre-OST0000 complete [ 7773.027227] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7773.028396] LDISKFS-fs (dm-0): recovery complete [ 7773.032118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.417608] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ca51b7b8c222e44 [ 7785.711022] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7789.652200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13473) [ 7800.048286] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7800.049391] LDISKFS-fs (dm-2): recovery complete [ 7800.051995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7801.813748] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7806.877815] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 00:21:51 (1763356911) [ 7811.540182] Lustre: Failing over lustre-OST0000 [ 7811.599837] Lustre: server umount lustre-OST0000 complete [ 7812.703208] Lustre: Failing over lustre-MDT0000 [ 7812.953317] Lustre: server umount lustre-MDT0000 complete [ 7825.535984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.824300] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7831.030572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13505) [ 7837.231064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7838.954724] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7839.623316] Lustre: lustre-OST0000: Denying connection for new client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7839.627495] Lustre: Skipped 1 previous similar message [ 7845.073241] Lustre: lustre-OST0000: Denying connection for new client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 7845.077152] Lustre: Skipped 1 previous similar message [ 7855.313384] Lustre: lustre-OST0000: Denying connection for new client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 7855.317274] Lustre: Skipped 1 previous similar message [ 7875.792935] Lustre: lustre-OST0000: Denying connection for new client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 7875.796798] Lustre: Skipped 3 previous similar messages [ 7909.000072] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7909.001561] Lustre: 255624:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 00d1f1e3-e590-4af2-bbc0-bbe6e1670853@ [ 7909.004282] Lustre: 255624:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7909.006138] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7909.010568] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7909.010634] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 7909.012517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13524 to 0x280000401:13545) [ 7909.012814] Lustre: Skipped 68 previous similar messages [ 7909.022284] Lustre: Skipped 21 previous similar messages [ 7912.932693] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 7917.775455] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 7919.838267] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 00:23:44 (1763357024) [ 7921.165407] Lustre: Failing over lustre-OST0000 [ 7921.191081] LustreError: 256959:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7921.192757] LustreError: 256959:0:(obd_class.h:479:obd_check_dev()) Skipped 163 previous similar messages [ 7921.207080] Lustre: server umount lustre-OST0000 complete [ 7921.633340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7921.636343] Lustre: Skipped 60 previous similar messages [ 7933.978696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7934.032842] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7934.034634] Lustre: Skipped 22 previous similar messages [ 7935.194349] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7935.196223] Lustre: Skipped 21 previous similar messages [ 7935.622220] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7939.404545] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 00:24:03 (1763357043) [ 7940.553826] Lustre: Failing over lustre-OST0000 [ 7942.647653] Lustre: server umount lustre-OST0000 complete [ 7955.007040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7956.576535] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7956.907909] LustreError: 258705:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7956.909718] LustreError: 258705:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 79 previous similar messages [ 7962.592139] Lustre: *** cfs_fail_loc=715, val=40*** [ 7973.079868] Lustre: lustre-OST0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 3 clients in recovery for 1:23 [ 7973.344092] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763357062/real 1763357062] req@ffff8a1cc960fb80 x1849005700978048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763357078 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 7973.349357] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [ 7979.488229] Lustre: *** cfs_fail_loc=715, val=40*** [ 7979.489170] Lustre: Skipped 2 previous similar messages [ 7988.433078] Lustre: lustre-OST0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [ 7988.436141] Lustre: Skipped 2 previous similar messages [ 7994.848248] Lustre: *** cfs_fail_loc=715, val=40*** [ 7994.849438] Lustre: Skipped 2 previous similar messages [ 7996.952065] LustreError: 258705:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7996.954020] LustreError: 258705:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 79 previous similar messages [ 7998.122203] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7998.584237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8001.645346] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 00:25:06 (1763357106) [ 8002.787498] Lustre: Failing over lustre-MDT0000 [ 8003.012424] Lustre: server umount lustre-MDT0000 complete [ 8015.434049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.692585] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8020.969139] LustreError: 260145:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8025.568189] Lustre: *** cfs_fail_loc=715, val=80*** [ 8025.569523] Lustre: Skipped 2 previous similar messages [ 8034.513051] Lustre: lustre-MDT0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 1:26 [ 8034.515962] Lustre: Skipped 2 previous similar messages [ 8037.344440] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8040.928182] Lustre: *** cfs_fail_loc=715, val=80*** [ 8040.929645] Lustre: Skipped 1 previous similar message [ 8050.896812] Lustre: lustre-MDT0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 8052.704587] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8057.312181] Lustre: *** cfs_fail_loc=715, val=80*** [ 8057.313252] Lustre: Skipped 1 previous similar message [ 8066.257189] Lustre: lustre-MDT0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8069.088535] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8075.232143] Lustre: *** cfs_fail_loc=715, val=80*** [ 8075.233374] Lustre: Skipped 2 previous similar messages [ 8082.641076] Lustre: lustre-MDT0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 0:38 [ 8085.472589] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8099.025031] Lustre: lustre-MDT0000: Client e56a79dd-caae-496b-b436-3e0ebc4c101c (at 192.168.202.31@tcp) reconnected, waiting for 2 clients in recovery for 0:21 [ 8100.832753] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8101.048075] LustreError: 260145:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8101.052683] Lustre: 260145:0:(ldlm_lib.c:2929:target_recovery_thread()) too long recovery - read logs [ 8101.055216] LustreError: dumping log to /tmp/lustre-log.1763357206.260145 [ 8101.111646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13518 to 0x2c0000401:13537) [ 8101.111651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13559 to 0x280000401:13577) [ 8102.314938] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8102.832348] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8106.234770] Lustre: DEBUG MARKER: == replay-single test complete, duration 7860 sec ======== 00:26:50 (1763357210) [ 8106.706688] Lustre: DEBUG MARKER: === replay-single: start cleanup 00:26:51 (1763357211) === [ 8109.204080] Lustre: DEBUG MARKER: === replay-single: finish cleanup 00:26:53 (1763357213) === [ 8109.838316] Lustre: Failing over lustre-MDT0000 [ 8110.068776] Lustre: server umount lustre-MDT0000 complete [ 8111.074553] LustreError: 256078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.078381] LustreError: 256078:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 219 previous similar messages [ 8124.987684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.023731] LustreError: MGC192.168.202.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.027045] LustreError: Skipped 7 previous similar messages [ 8125.127137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.129332] Lustre: Skipped 19 previous similar messages [ 8126.780553] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8130.554761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13518 to 0x2c0000401:13569) [ 8130.554791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13559 to 0x280000401:13609) [ 8131.624863] Lustre: DEBUG MARKER: oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8132.134430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8140.769128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8140.769494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8140.771064] LustreError: Skipped 12 previous similar messages [ 8140.773601] Lustre: Skipped 21 previous similar messages [ 8145.425216] Lustre: server umount lustre-MDT0000 complete [ 8147.899564] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763357253 with bad export cookie 8981615249306328168 [ 8147.902272] LustreError: 6507:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8148.012670] Lustre: server umount lustre-MDT0001 complete [ 8161.281549] Lustre: server umount lustre-OST0000 complete [ 8174.184126] Lustre: server umount lustre-OST0001 complete [ 8179.075577] Lustre: DEBUG MARKER: oleg231-server.virtnet: executing unload_modules_local [ 8180.045284] Key type lgssc unregistered [ 8180.178338] LNet: 264715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8180.180462] LNetError: 264715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8180.186525] LNet: Removed LNI 192.168.202.131@tcp [ 8180.512122] Key type .llcrypt unregistered [ 8180.513107] Key type ._llcrypt unregistered