[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 454792890 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002379] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006712] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012062] *** VALIDATE selinux *** [ 0.020345] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025409] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] *** VALIDATE tmpfs *** [ 0.029377] *** VALIDATE proc *** [ 0.030185] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033121] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035021] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.039216] debug: unmapping init [mem 0xffffffff9b859000-0xffffffff9b860fff] [ 0.041826] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042527] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047020] ... max period: 00007fffffffffff [ 0.048015] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050296] rcu: Hierarchical SRCU implementation. [ 0.052172] smp: Bringing up secondary CPUs ... [ 0.053415] x86: Booting SMP configuration: [ 0.054029] .... node #0, CPUs: #1 #2 #3 [ 0.057481] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.059846] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130852] node 0 deferred pages initialised in 69ms [ 0.132092] devtmpfs: initialized [ 0.133097] x86/mm: Memory block size: 128MB [ 0.135683] gcov: version magic: 0x41383552 [ 0.136652] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141269] pinctrl core: initialized pinctrl subsystem [ 0.142187] [ 0.142691] ************************************************************* [ 0.143017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144014] ** ** [ 0.145012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146015] ** ** [ 0.147013] ** This means that this kernel is built to expose internal ** [ 0.148013] ** IOMMU data structures, which may compromise security on ** [ 0.149011] ** your system. ** [ 0.150013] ** ** [ 0.151011] ** If you see this message and you are not debugging the ** [ 0.152025] ** kernel, report this immediately to your vendor! ** [ 0.153014] ** ** [ 0.154013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155015] ************************************************************* [ 0.156578] NET: Registered protocol family 16 [ 0.157420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160381] cpuidle: using governor menu [ 0.162895] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164494] PCI: Using configuration type 1 for base access [ 0.166125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178123] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181112] cryptd: max_cpu_qlen set to 1000 [ 0.183271] ACPI: Added _OSI(Module Device) [ 0.184000] ACPI: Added _OSI(Processor Device) [ 0.184011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.189600] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.194353] ACPI: Interpreter enabled [ 0.195043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.196011] ACPI: Using IOAPIC for interrupt routing [ 0.197103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.198249] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.206359] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.208016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210968] acpiphp: Slot [2] registered [ 0.211069] acpiphp: Slot [5] registered [ 0.211843] acpiphp: Slot [6] registered [ 0.212058] acpiphp: Slot [7] registered [ 0.213061] acpiphp: Slot [8] registered [ 0.214042] acpiphp: Slot [9] registered [ 0.214961] acpiphp: Slot [10] registered [ 0.216123] acpiphp: Slot [3] registered [ 0.218122] acpiphp: Slot [4] registered [ 0.220115] acpiphp: Slot [11] registered [ 0.221058] acpiphp: Slot [12] registered [ 0.222029] acpiphp: Slot [13] registered [ 0.223108] acpiphp: Slot [14] registered [ 0.224090] acpiphp: Slot [15] registered [ 0.226099] acpiphp: Slot [16] registered [ 0.227107] acpiphp: Slot [17] registered [ 0.228098] acpiphp: Slot [18] registered [ 0.230116] acpiphp: Slot [19] registered [ 0.231109] acpiphp: Slot [20] registered [ 0.232083] acpiphp: Slot [21] registered [ 0.234087] acpiphp: Slot [22] registered [ 0.235079] acpiphp: Slot [23] registered [ 0.237099] acpiphp: Slot [24] registered [ 0.238101] acpiphp: Slot [25] registered [ 0.240097] acpiphp: Slot [26] registered [ 0.241133] acpiphp: Slot [27] registered [ 0.243099] acpiphp: Slot [28] registered [ 0.244102] acpiphp: Slot [29] registered [ 0.246129] acpiphp: Slot [30] registered [ 0.249112] acpiphp: Slot [31] registered [ 0.250048] PCI host bridge to bus 0000:00 [ 0.252010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.255022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.257015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.258017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.260017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.263766] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.266047] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.279058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.281019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.283024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.286020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.288404] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.289782] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.292032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.296045] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.314020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.319014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.324012] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.330016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.335016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.360019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.369595] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.375022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.380017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.393017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.399485] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.404012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.409029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.421017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.431799] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.439016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.447016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.465025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.475336] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.483017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.491016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.516020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.528333] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.533014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.539015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.555019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.565649] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.568425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.570356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.572345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.575202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.579094] iommu: Default domain type: Passthrough [ 0.580373] SCSI subsystem initialized [ 0.581113] ACPI: bus type USB registered [ 0.583095] usbcore: registered new interface driver usbfs [ 0.584090] usbcore: registered new interface driver hub [ 0.585079] usbcore: registered new device driver usb [ 0.587116] pps_core: LinuxPPS API ver. 1 registered [ 0.588009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.590067] PTP clock support registered [ 0.592110] EDAC MC: Ver: 3.0.0 [ 0.593550] PCI: Using ACPI for IRQ routing [ 0.595717] NetLabel: Initializing [ 0.596013] NetLabel: domain hash size = 128 [ 0.597009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.599073] NetLabel: unlabeled traffic allowed by default [ 0.600135] vgaarb: loaded [ 0.601246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.603010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.609118] clocksource: Switched to clocksource kvm-clock [ 0.712285] VFS: Disk quotas dquot_6.6.0 [ 0.713908] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.716219] *** VALIDATE ramfs *** [ 0.717580] *** VALIDATE hugetlbfs *** [ 0.719184] pnp: PnP ACPI init [ 0.721652] pnp: PnP ACPI: found 6 devices [ 0.742067] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.745570] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.747165] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.748654] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.750557] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.752833] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.755788] NET: Registered protocol family 2 [ 0.758229] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.762971] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.766631] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.771425] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.774774] TCP: Hash tables configured (established 65536 bind 65536) [ 0.777816] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.780503] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.782658] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.784841] NET: Registered protocol family 1 [ 0.787078] RPC: Registered named UNIX socket transport module. [ 0.788674] RPC: Registered udp transport module. [ 0.789894] RPC: Registered tcp transport module. [ 0.791326] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.792949] NET: Registered protocol family 44 [ 0.794107] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.795600] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.797127] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.798720] PCI: CLS 0 bytes, default 64 [ 0.799901] Unpacking initramfs... [ 2.163091] debug: unmapping init [mem 0xffff8bd87cc54000-0xffff8bd87ffbffff] [ 2.165664] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.167072] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.168893] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.641205] Initialise system trusted keyrings [ 2.642889] Key type blacklist registered [ 2.644777] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.653915] zbud: loaded [ 2.657244] *** VALIDATE nfs *** [ 2.658501] *** VALIDATE nfs4 *** [ 2.660045] pstore: using deflate compression [ 2.663627] Platform Keyring initialized [ 2.774763] NET: Registered protocol family 38 [ 2.776872] Key type asymmetric registered [ 2.778876] Asymmetric key parser 'x509' registered [ 2.781327] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.785061] io scheduler mq-deadline registered [ 2.786767] io scheduler kyber registered [ 2.790331] io scheduler bfq registered [ 2.793305] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.796762] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.799620] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.802810] ACPI: Power Button [PWRF] [ 2.808673] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.814639] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.825474] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.831892] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.845146] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.871058] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.896771] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.900941] Non-volatile memory driver v1.3 [ 2.902381] Linux agpgart interface v0.103 [ 2.936428] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 2.939958] vda: detected capacity change from 0 to 34836480 [ 2.980656] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.982922] vdb: detected capacity change from 0 to 1073741824 [ 2.997704] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.000623] vdc: detected capacity change from 0 to 2621440000 [ 3.014151] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.016431] vdd: detected capacity change from 0 to 2621440000 [ 3.030136] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.032312] vde: detected capacity change from 0 to 4294967296 [ 3.048448] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.051239] vdf: detected capacity change from 0 to 4294967296 [ 3.058782] libphy: Fixed MDIO Bus: probed [ 3.076840] usbcore: registered new interface driver usbserial_generic [ 3.079601] usbserial: USB Serial support registered for generic [ 3.082108] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.086496] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.088524] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.090935] mousedev: PS/2 mouse device common for all mice [ 3.094361] rtc_cmos 00:05: RTC can wake from S4 [ 3.094421] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.097742] rtc_cmos 00:05: registered as rtc0 [ 3.101429] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.102780] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.104807] intel_pstate: CPU model not supported [ 3.110865] hid: raw HID events driver (C) Jiri Kosina [ 3.110990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.112865] usbcore: registered new interface driver usbhid [ 3.117567] usbhid: USB HID core driver [ 3.119034] drop_monitor: Initializing network drop monitor service [ 3.121135] Initializing XFRM netlink socket [ 3.122812] NET: Registered protocol family 10 [ 3.125710] Segment Routing with IPv6 [ 3.127617] NET: Registered protocol family 17 [ 3.129725] mpls_gso: MPLS GSO support [ 3.135990] RAS: Correctable Errors collector initialized. [ 3.138658] AVX version of gcm_enc/dec engaged. [ 3.140810] AES CTR mode by8 optimization enabled [ 3.206057] sched_clock: Marking stable (3206007955, 0)->(4091648021, -885640066) [ 3.209538] registered taskstats version 1 [ 3.211679] Loading compiled-in X.509 certificates [ 3.213555] zswap: loaded using pool lzo/zbud [ 3.237997] Key type big_key registered [ 3.249312] Key type encrypted registered [ 3.251046] ima: No TPM chip found, activating TPM-bypass! [ 3.252922] ima: Allocated hash algorithm: sha1 [ 3.254925] ima: No architecture policies found [ 3.256911] evm: Initialising EVM extended attributes: [ 3.259451] evm: security.selinux [ 3.260716] evm: security.ima [ 3.261877] evm: security.capability [ 3.262877] evm: HMAC attrs: 0x1 [ 3.264656] rtc_cmos 00:05: setting system clock to 2026-04-14 14:52:23 UTC (1776178343) [ 3.270452] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 3.274567] debug: unmapping init [mem 0xffffffff9b582000-0xffffffff9b858fff] [ 3.284062] Write protecting the kernel read-only data: 28672k [ 3.286883] debug: unmapping init [mem 0xffffffff99c03000-0xffffffff99dfffff] [ 3.289645] debug: unmapping init [mem 0xffffffff9a514000-0xffffffff9a5fffff] [ 3.325826] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.334553] systemd[1]: Detected virtualization kvm. [ 3.336593] systemd[1]: Detected architecture x86-64. [ 3.338651] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.363045] systemd[1]: No hostname configured. [ 3.364524] systemd[1]: Set hostname to . [ 3.366437] random: systemd: uninitialized urandom read (16 bytes read) [ 3.368958] systemd[1]: Initializing machine ID from random generator. [ 3.406710] random: ln: uninitialized urandom read (6 bytes read) [ 3.494944] random: systemd: uninitialized urandom read (16 bytes read) [ 3.497128] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.500682] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.504421] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.114983] device-mapper: uevent: version 1.0.3 [ 4.117124] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.497649] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.864350] virtio_net virtio0 ens2: renamed from eth0 [ 4.932596] scsi host0: ata_piix [ 4.959486] scsi host1: ata_piix [ 4.960803] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.962865] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.341364] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.767616] random: crng init done [ 9.768708] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.205891] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.404450] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.707535] SELinux: Disabled at runtime. [ 11.773040] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.780783] systemd[1]: Detected virtualization kvm. [ 11.782615] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.310752] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.314427] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.319401] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.325281] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.329941] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.337407] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.341459] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK [[ 12.552168] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.855315] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.160287] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.277430] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.415412] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.429672] EDAC sbridge: Ver: 1.1.2 [ 15.330045] Key type dns_resolver registered [ 15.632145] NFS: Registering the id_resolver key type [ 15.634074] Key type id_resolver registered [ 15.635669] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 46.144343] hrtimer: interrupt took 2724944 ns [ 52.919163] libcfs: loading out-of-tree module taints kernel. [ 53.095262] alg: No test for adler32 (adler32-zlib) [ 53.866686] Key type ._llcrypt registered [ 53.869993] Key type .llcrypt registered [ 54.106807] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 74.552900] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 76.374164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 77.471676] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 78.274082] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 78.280239] LNet: Accept secure, port 988 [ 80.079159] Key type lgssc registered [ 81.845926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.613177] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 103.167967] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.918695] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.719363] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.075272] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.918541] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 154.469701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.549187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.784715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 155.818537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 155.931192] Lustre: lustre-MDT0000: new disk, initializing [ 156.003558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.023743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.350930] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 171.848970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.942917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.036315] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 172.052513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 172.055016] Lustre: Skipped 1 previous similar message [ 172.137261] Lustre: lustre-MDT0001: new disk, initializing [ 172.182901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.201321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.208816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.440792] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 193.585527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 193.668262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.678345] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 193.892575] Lustre: lustre-OST0000: new disk, initializing [ 193.895470] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 193.936104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.570582] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 200.216167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.226401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.523353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.602671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.608437] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 209.701957] Lustre: lustre-OST0001: new disk, initializing [ 209.709096] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 209.798682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.996951] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 215.124302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.131262] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 222.955836] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.063478] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 247.588905] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 258.445172] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 263.966423] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 267.604449] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 270.508126] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 271.823303] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 10:56:50 EDT 2026 [ 279.522911] Lustre: DEBUG MARKER: excepting tests: 59 [ 284.432250] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing check_config_client /mnt/lustre [ 301.695202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.152590] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 307.341490] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 310.228394] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 10:57:29 (1776178649) [ 316.031571] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 317.557244] Lustre: Failing over lustre-MDT0000 [ 317.702518] Lustre: server umount lustre-MDT0000 complete [ 317.926234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.932759] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.941823] Lustre: Skipped 3 previous similar messages [ 317.961369] LustreError: Skipped 3 previous similar messages [ 321.902857] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.040715] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.066872] LustreError: Skipped 3 previous similar messages [ 326.998146] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.719186] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178663/real 1776178663] req@000000008ff073f5 x1862458060885504/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178670 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 330.739585] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.169084] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.182120] LustreError: Skipped 9 previous similar messages [ 339.506750] LDISKFS-fs (dm-0): recovery complete [ 339.509218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.433155] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.443821] LustreError: Skipped 11 previous similar messages [ 347.107028] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af04eae to 0x2b81c8ee0af05561 [ 347.119778] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 347.256472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.728034] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 351.896357] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 352.765264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 352.855063] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 361.040466] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 362.386659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 370.410577] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 10:58:29 (1776178709) [ 372.242481] Lustre: Failing over lustre-OST0000 [ 372.269983] Lustre: server umount lustre-OST0000 complete [ 373.111144] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.125440] LustreError: Skipped 9 previous similar messages [ 373.216818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.229496] Lustre: Skipped 1 previous similar message [ 390.843378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 390.986150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.319530] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 392.733627] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 392.733762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.202.132@tcp (at 0@lo) [ 392.752702] Lustre: Skipped 3 previous similar messages [ 392.765062] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 394.671191] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 401.588446] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 402.726904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 408.861449] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 10:59:07 (1776178747) [ 414.749702] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 416.130602] Lustre: Failing over lustre-MDT0000 [ 416.217035] Lustre: server umount lustre-MDT0000 complete [ 416.736679] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.737929] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.745706] Lustre: Skipped 3 previous similar messages [ 416.750969] LustreError: Skipped 13 previous similar messages [ 429.535197] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178762/real 1776178762] req@000000009ac7c668 x1862458060914944/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178769 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 429.550173] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.869621] LDISKFS-fs (dm-0): recovery complete [ 437.876236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.923750] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af05561 to 0x2b81c8ee0af05f8d [ 445.949957] LustreError: 17464:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.132@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000cf65a6b5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 445.977920] Lustre: MGC192.168.202.132@tcp: Connection restored to 192.168.202.132@tcp (at 0@lo) [ 445.991238] Lustre: Skipped 1 previous similar message [ 446.417048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.884885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 451.556672] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 451.580485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 453.037841] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:39 [ 458.076945] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:34 [ 463.194305] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:29 [ 468.318682] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:24 [ 473.436146] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:19 [ 483.675709] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 483.690058] Lustre: Skipped 1 previous similar message [ 504.152462] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 504.175596] Lustre: Skipped 3 previous similar messages [ 539.992649] Lustre: lustre-MDT0000: Denying connection for new client 5c0e68b7-44cb-46c8-a6f8-9e8bb7a7690d (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:13 [ 540.000620] Lustre: Skipped 6 previous similar messages [ 553.000326] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 553.008893] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 553.031356] Lustre: lustre-MDT0000: Recovery over after 1:42, of 2 clients 1 recovered and 1 was evicted. [ 553.031449] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 553.047209] Lustre: Skipped 2 previous similar messages [ 553.059109] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:97 [ 553.059698] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 567.376399] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:01:45 (1776178905) [ 576.079716] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 578.057249] Lustre: Failing over lustre-MDT0000 [ 578.202599] Lustre: server umount lustre-MDT0000 complete [ 578.528387] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 578.540436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.577129] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.599305] LustreError: Skipped 29 previous similar messages [ 586.591153] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178919/real 1776178919] req@00000000e3d0a416 x1862458060950464/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178926 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 586.607601] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.354024] LDISKFS-fs (dm-0): recovery complete [ 607.361934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.494613] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af05f8d to 0x2b81c8ee0af06399 [ 619.508865] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 619.723018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.833359] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.122364] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 626.332336] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 629.075133] Lustre: lustre-MDT0000: Denying connection for new client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:37 [ 629.088487] Lustre: Skipped 2 previous similar messages [ 727.000318] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 727.006416] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 727.022895] Lustre: lustre-MDT0000: Recovery over after 1:42, of 2 clients 1 recovered and 1 was evicted. [ 727.025134] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 727.031802] Lustre: Skipped 3 previous similar messages [ 727.058723] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:97 [ 727.068522] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:129 [ 740.073903] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:04:38 (1776179078) [ 747.321743] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 749.253284] Lustre: Failing over lustre-MDT0000 [ 749.428793] Lustre: server umount lustre-MDT0000 complete [ 751.993875] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 752.018938] LustreError: Skipped 44 previous similar messages [ 752.612488] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 752.639784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.651865] Lustre: Skipped 3 previous similar messages [ 760.287167] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179093/real 1776179093] req@000000005745f41f x1862458060986752/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179100 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 760.318676] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.411363] LDISKFS-fs (dm-0): recovery complete [ 771.415367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.697695] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af06399 to 0x2b81c8ee0af06815 [ 778.014766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.075970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 779.120390] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 782.492386] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 783.420869] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 783.503380] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:161 [ 783.505431] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:129 [ 792.418513] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 794.308269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 804.521715] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:05:42 (1776179142) [ 813.169090] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 815.146034] Lustre: Failing over lustre-MDT0000 [ 815.304213] Lustre: server umount lustre-MDT0000 complete [ 819.170582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.170677] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 819.196909] Lustre: Skipped 6 previous similar messages [ 839.670281] LDISKFS-fs (dm-0): recovery complete [ 839.672700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.824337] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 839.828749] LustreError: 11-0: MGC192.168.202.132@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 839.835892] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af06815 to 0x2b81c8ee0af06d16 [ 839.858325] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 839.866592] Lustre: Skipped 5 previous similar messages [ 840.148519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 842.189263] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 845.488737] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 845.568927] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:161 [ 845.569686] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 846.547730] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 857.996781] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 860.135614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 871.377936] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:06:49 (1776179209) [ 879.274883] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 880.891577] Lustre: Failing over lustre-MDT0000 [ 881.026512] Lustre: server umount lustre-MDT0000 complete [ 881.131899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.147686] Lustre: Skipped 2 previous similar messages [ 888.287209] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179221/real 1776179221] req@0000000014f570b5 x1862458061024128/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179228 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 888.306629] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 905.802664] LDISKFS-fs (dm-0): recovery complete [ 905.806132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.059676] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af06d16 to 0x2b81c8ee0af0726b [ 921.231595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 921.235093] Lustre: Skipped 1 previous similar message [ 921.276792] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 922.971311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 925.815110] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 926.875423] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 926.944975] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:193 [ 926.948169] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:225 [ 935.361566] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 937.100372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 944.164155] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:08:03 (1776179283) [ 952.994223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 955.523256] Lustre: Failing over lustre-MDT0000 [ 955.756022] Lustre: server umount lustre-MDT0000 complete [ 957.408966] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.422977] Lustre: Skipped 3 previous similar messages [ 964.575427] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179297/real 1776179297] req@00000000f4a822e4 x1862458061044096/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179304 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 964.609094] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 979.606144] LDISKFS-fs (dm-0): recovery complete [ 979.607708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.974461] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af0726b to 0x2b81c8ee0af0777a [ 980.980052] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 980.985524] Lustre: Skipped 9 previous similar messages [ 981.305823] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 982.359559] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 985.990179] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 986.714257] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 986.781504] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:225 [ 986.782358] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 995.425694] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 996.812120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1003.999526] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:09:02 (1776179342) [ 1009.309678] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1012.196690] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.207211] LustreError: Skipped 155 previous similar messages [ 1019.231166] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179352/real 1776179352] req@000000001637fe7b x1862458061060992/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179359 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1019.258590] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.913251] LDISKFS-fs (dm-0): recovery complete [ 1030.916937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.779222] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af0777a to 0x2b81c8ee0af07cb3 [ 1037.025930] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1040.881529] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1042.507848] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:257 [ 1042.511715] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:289 [ 1048.308994] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1049.520812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1054.859328] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:09:54 (1776179394) [ 1055.644949] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1055.653342] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1055.658911] LustreError: 6265:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000011652e55 x1862458040985920/t38654705666(0) o35->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:376/0 lens 392/456 e 0 to 0 dl 1776179401 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 1061.626979] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1062.764691] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 1062.786315] Lustre: 6265:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000007c762bf4 x1862458040985920/t38654705666(0) o35->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:384/0 lens 392/456 e 0 to 0 dl 1776179409 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 1063.142123] Lustre: Failing over lustre-MDT0000 [ 1063.145314] Lustre: Skipped 1 previous similar message [ 1063.326720] Lustre: server umount lustre-MDT0000 complete [ 1063.329599] Lustre: Skipped 1 previous similar message [ 1068.000420] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.032648] Lustre: Skipped 7 previous similar messages [ 1075.103187] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179408/real 1776179408] req@000000005dc15927 x1862458061078272/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179415 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1083.938532] LDISKFS-fs (dm-0): recovery complete [ 1083.941884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.758712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1093.642120] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1093.654243] Lustre: Skipped 1 previous similar message [ 1095.955140] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1098.269863] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1098.276168] Lustre: Skipped 1 previous similar message [ 1098.298065] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:321 [ 1098.315975] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:289 [ 1103.326382] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1104.847703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1111.987519] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:10:51 (1776179451) [ 1118.977397] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1130.978994] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.988494] LustreError: Skipped 1 previous similar message [ 1143.288780] LDISKFS-fs (dm-0): recovery complete [ 1143.291546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.386939] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af08198 to 0x2b81c8ee0af08645 [ 1148.398035] Lustre: Skipped 1 previous similar message [ 1148.599121] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1152.099137] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1154.088681] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:321 [ 1154.091357] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:353 [ 1159.562625] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1160.886509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1167.840101] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:11:46 (1776179506) [ 1174.736938] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1175.616412] Lustre: *** cfs_fail_loc=114, val=0*** [ 1186.703094] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179519/real 1776179519] req@0000000026f34460 x1862458061112064/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179526 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1186.721361] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1202.012444] LDISKFS-fs (dm-0): recovery complete [ 1202.016508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.523547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.535935] Lustre: Skipped 4 previous similar messages [ 1207.375935] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1208.974490] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:353 [ 1208.977753] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:385 [ 1218.442902] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1221.110969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1231.854618] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:12:50 (1776179570) [ 1240.190827] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1241.051803] Lustre: *** cfs_fail_loc=128, val=0*** [ 1243.577258] Lustre: Failing over lustre-MDT0000 [ 1243.589200] Lustre: Skipped 2 previous similar messages [ 1243.806798] Lustre: server umount lustre-MDT0000 complete [ 1243.809746] Lustre: Skipped 2 previous similar messages [ 1244.642800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.646454] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1244.679212] Lustre: Skipped 10 previous similar messages [ 1270.280753] LDISKFS-fs (dm-0): recovery complete [ 1270.292094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.600297] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af08b2a to 0x2b81c8ee0af08fde [ 1284.608665] Lustre: Skipped 1 previous similar message [ 1284.612025] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 1284.616031] Lustre: Skipped 24 previous similar messages [ 1284.779382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1284.789421] Lustre: Skipped 1 previous similar message [ 1284.949701] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1284.957879] Lustre: Skipped 2 previous similar messages [ 1288.749169] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1290.268961] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 1290.285560] Lustre: Skipped 2 previous similar messages [ 1290.344301] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:385 [ 1290.347868] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:417 [ 1296.787768] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1297.899298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1304.662816] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:14:03 (1776179643) [ 1310.547691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1315.810047] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.975775] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179656/real 1776179656] req@00000000c46746aa x1862458061150336/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179663 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 1323.002348] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1323.006776] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.016722] LustreError: Skipped 2 previous similar messages [ 1335.884469] LDISKFS-fs (dm-0): recovery complete [ 1335.887467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.541499] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1346.290323] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 1346.291038] Lustre: lustre-OST0000: deleting orphan objects from 0x0:423 to 0x0:449 [ 1354.514365] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1356.313353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1364.059539] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:15:03 (1776179703) [ 1370.938985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1395.535940] LDISKFS-fs (dm-0): recovery complete [ 1395.542636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.011722] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1407.255836] Lustre: lustre-OST0001: deleting orphan objects from 0x0:423 to 0x0:449 [ 1407.257394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 1414.867906] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1416.866959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1424.019753] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:16:02 (1776179762) [ 1430.389709] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1459.139973] LDISKFS-fs (dm-0): recovery complete [ 1459.149478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.732918] LustreError: 43858:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.132@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000008c23dc8e) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1462.114780] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (not set up) [ 1462.301651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1462.313280] Lustre: Skipped 2 previous similar messages [ 1468.450940] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1471.093331] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:577 [ 1471.100274] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:609 [ 1477.958197] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1479.960804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1499.975433] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:17:19 (1776179839) [ 1507.181555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1508.770338] Lustre: Failing over lustre-MDT0000 [ 1508.774154] Lustre: Skipped 3 previous similar messages [ 1508.886377] Lustre: server umount lustre-MDT0000 complete [ 1508.889551] Lustre: Skipped 3 previous similar messages [ 1511.907879] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1511.917643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.937033] Lustre: Skipped 14 previous similar messages [ 1525.591309] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.608230] LustreError: Skipped 342 previous similar messages [ 1528.045527] LDISKFS-fs (dm-0): recovery complete [ 1528.049966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.088754] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1533.511723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:641 [ 1533.513465] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:609 [ 1539.273535] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1540.727685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1549.113628] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:18:08 (1776179888) [ 1555.925965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1579.301847] LDISKFS-fs (dm-0): recovery complete [ 1579.306848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.595592] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af12e92 to 0x2b81c8ee0af133cb [ 1583.611962] Lustre: Skipped 4 previous similar messages [ 1584.983818] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1584.996471] Lustre: Skipped 4 previous similar messages [ 1587.688806] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1589.283073] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1589.286046] Lustre: Skipped 4 previous similar messages [ 1589.322850] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:641 [ 1589.323722] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:673 [ 1596.996942] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1598.475886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1606.356630] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:19:05 (1776179945) [ 1612.721709] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1614.837914] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1614.848920] LustreError: Skipped 1 previous similar message [ 1627.087163] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179960/real 1776179960] req@00000000b00b4366 x1862458061275008/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179967 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 1627.124920] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1627.140362] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1627.158761] LustreError: Skipped 4 previous similar messages [ 1642.022709] LDISKFS-fs (dm-0): recovery complete [ 1642.030236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.751977] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1650.266185] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:705 [ 1650.267705] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:673 [ 1657.579797] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1658.947895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1667.121428] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:20:05 (1776180005) [ 1673.753060] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1698.595056] LDISKFS-fs (dm-0): recovery complete [ 1698.597626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.196848] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1711.298609] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:705 [ 1711.299581] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:737 [ 1719.471854] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1721.323666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1729.781248] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:21:08 (1776180068) [ 1737.193228] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1760.751855] LDISKFS-fs (dm-0): recovery complete [ 1760.754542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.537670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.540557] Lustre: Skipped 8 previous similar messages [ 1766.578220] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1766.581396] Lustre: Skipped 4 previous similar messages [ 1770.656241] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1772.154345] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:769 [ 1772.156158] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:737 [ 1780.730458] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1782.775419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1790.649230] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:22:09 (1776180129) [ 1797.480248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1802.727447] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1802.737272] LustreError: Skipped 1 previous similar message [ 1820.398887] LDISKFS-fs (dm-0): recovery complete [ 1820.402467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1826.299207] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 1826.307933] Lustre: Skipped 44 previous similar messages [ 1831.902578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1832.090300] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:801 [ 1832.091467] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:769 [ 1842.736577] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1844.435904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1852.169740] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:23:11 (1776180191) [ 1858.156851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1883.340407] LDISKFS-fs (dm-0): recovery complete [ 1883.343594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.000462] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:833 [ 1889.002824] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1889.168732] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1898.496826] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1899.672849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1906.600591] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:24:05 (1776180245) [ 1912.674556] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1914.851741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.069314] LDISKFS-fs (dm-0): recovery complete [ 1939.073900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1944.049621] LustreError: 60223:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.132@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000de6b95c9) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1948.318111] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1949.736642] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:865 [ 1949.739168] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:833 [ 1957.886387] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1959.476766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1968.871950] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:25:07 (1776180307) [ 1979.112203] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1985.507606] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1985.516657] LustreError: Skipped 2 previous similar messages [ 2006.821567] LDISKFS-fs (dm-0): recovery complete [ 2006.825630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.406553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:867 to 0x0:897 [ 2016.407108] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:865 [ 2017.466894] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2029.773280] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2031.599912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2041.303666] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:26:19 (1776180379) [ 2049.920190] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2052.472515] Lustre: Failing over lustre-MDT0000 [ 2052.483719] Lustre: Skipped 8 previous similar messages [ 2052.914723] Lustre: server umount lustre-MDT0000 complete [ 2052.916693] Lustre: Skipped 8 previous similar messages [ 2057.187736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.200418] Lustre: Skipped 35 previous similar messages [ 2075.955306] LDISKFS-fs (dm-0): recovery complete [ 2075.958280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.580804] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2087.545742] Lustre: lustre-OST0000: deleting orphan objects from 0x0:899 to 0x0:929 [ 2087.548611] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:897 [ 2097.109678] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.857576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2107.187814] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:27:26 (1776180446) [ 2114.847858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2126.641667] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.658680] LustreError: Skipped 364 previous similar messages [ 2139.531958] LDISKFS-fs (dm-0): recovery complete [ 2139.534033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.696095] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af15f29 to 0x2b81c8ee0af16454 [ 2142.703090] Lustre: Skipped 8 previous similar messages [ 2142.711420] LustreError: 66349:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.132@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000040ce4963) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2143.092124] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2143.100436] Lustre: Skipped 8 previous similar messages [ 2147.238562] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2148.459106] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2148.468560] Lustre: Skipped 8 previous similar messages [ 2148.522491] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:929 [ 2148.523313] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:961 [ 2156.679383] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2158.321578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2167.148601] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:28:25 (1776180505) [ 2174.445700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2186.655204] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180519/real 1776180519] req@00000000954bc9c9 x1862458061440320/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180526 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2186.689718] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2186.700770] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2186.712254] LustreError: Skipped 8 previous similar messages [ 2201.082191] LDISKFS-fs (dm-0): recovery complete [ 2201.097013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.675517] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2208.832803] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:993 [ 2208.833810] Lustre: lustre-OST0001: deleting orphan objects from 0x0:931 to 0x0:961 [ 2215.791748] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2217.246865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2224.116888] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:29:23 (1776180563) [ 2229.294832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2254.133340] LDISKFS-fs (dm-0): recovery complete [ 2254.136818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.863086] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2263.640203] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:1025 [ 2263.640935] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 2270.905243] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2272.375578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2279.932911] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:30:18 (1776180618) [ 2287.860240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2294.241402] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.249572] LustreError: Skipped 3 previous similar messages [ 2312.199168] LDISKFS-fs (dm-0): recovery complete [ 2312.203769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.116318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2318.123719] Lustre: Skipped 8 previous similar messages [ 2322.757485] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2323.631966] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2323.644551] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2332.329430] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.028774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2342.568277] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:31:21 (1776180681) [ 2349.717275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2374.158102] LDISKFS-fs (dm-0): recovery complete [ 2374.163407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.513390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.523666] Lustre: Skipped 9 previous similar messages [ 2383.504308] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2385.012540] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2385.013305] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 2391.875673] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2393.063775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2399.755733] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:32:18 (1776180738) [ 2407.041450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2430.704773] LDISKFS-fs (dm-0): recovery complete [ 2430.709513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.049805] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 2435.055922] Lustre: Skipped 49 previous similar messages [ 2439.263669] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2440.801852] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1089 [ 2440.807068] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2447.089679] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2448.358897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2455.735523] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:33:14 (1776180794) [ 2459.652932] Lustre: 77569:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c3202855-adee-42cc-8c9b-dd19f330dea1 at adminstrative request [ 2485.296203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.563194] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1121 [ 2495.570562] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2495.672851] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2507.878780] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2509.453315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2515.170884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2522.886413] Lustre: DEBUG MARKER: before 2844, after 2844 [ 2529.680278] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:34:28 (1776180868) [ 2530.678053] Lustre: 79604:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c3202855-adee-42cc-8c9b-dd19f330dea1 at adminstrative request [ 2538.457097] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:34:37 (1776180877) [ 2544.824242] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2546.669600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.207380] LDISKFS-fs (dm-0): recovery complete [ 2572.209888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.673415] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2581.677509] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1124 to 0x0:1153 [ 2581.978327] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2593.774366] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2595.527719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2605.265489] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:35:44 (1776180944) [ 2611.781300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2613.597648] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 2613.608270] Lustre: Skipped 3 previous similar messages [ 2636.980336] LDISKFS-fs (dm-0): recovery complete [ 2636.986545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.761783] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2646.612056] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2646.614455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2655.438799] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2657.086915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2664.894737] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:36:43 (1776181003) [ 2671.848797] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2673.674389] Lustre: Failing over lustre-MDT0000 [ 2673.676238] Lustre: Skipped 9 previous similar messages [ 2673.921161] Lustre: server umount lustre-MDT0000 complete [ 2673.932240] Lustre: Skipped 9 previous similar messages [ 2677.217825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.239199] Lustre: Skipped 42 previous similar messages [ 2698.554294] LDISKFS-fs (dm-0): recovery complete [ 2698.557794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.446232] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2706.577261] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2706.580092] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2714.375183] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2715.709157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2722.788890] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:37:41 (1776181061) [ 2729.507128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2732.005684] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.019164] LustreError: Skipped 346 previous similar messages [ 2753.181144] LDISKFS-fs (dm-0): recovery complete [ 2753.184989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.701798] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af1959a to 0x2b81c8ee0af19b04 [ 2761.712339] Lustre: Skipped 9 previous similar messages [ 2763.488204] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2763.497105] Lustre: Skipped 9 previous similar messages [ 2766.476648] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2767.381931] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2767.394382] Lustre: Skipped 9 previous similar messages [ 2767.439264] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2767.439721] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2776.111510] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2777.796652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2786.023307] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:38:44 (1776181124) [ 2794.462978] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2805.087491] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181138/real 1776181138] req@00000000e7f2a101 x1862458061625664/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181145 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 2805.120278] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2805.134911] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2805.157718] LustreError: Skipped 9 previous similar messages [ 2818.474560] LDISKFS-fs (dm-0): recovery complete [ 2818.479577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.878656] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2828.387211] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2828.391768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1313 [ 2837.184995] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2838.573306] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2846.883661] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:39:45 (1776181185) [ 2853.523050] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2858.979169] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2858.985403] LustreError: Skipped 4 previous similar messages [ 2877.958347] LDISKFS-fs (dm-0): recovery complete [ 2877.962252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.124661] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2889.296054] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2889.297722] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2896.912647] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2898.471625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2904.900531] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:40:44 (1776181244) [ 2911.801887] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2933.056684] LDISKFS-fs (dm-0): recovery complete [ 2933.062454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.513847] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2939.519541] Lustre: Skipped 9 previous similar messages [ 2943.433641] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2945.113225] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2945.113256] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2950.749225] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2951.954860] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2959.082346] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:41:38 (1776181298) [ 2965.497721] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2987.266067] LDISKFS-fs (dm-0): recovery complete [ 2987.268912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.860852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.864977] Lustre: Skipped 9 previous similar messages [ 2998.519427] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3000.489737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 3000.490460] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 3009.036649] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.921413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3020.481612] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:42:39 (1776181359) [ 3028.121514] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3053.926092] LDISKFS-fs (dm-0): recovery complete [ 3053.937100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.082702] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 3055.095694] Lustre: Skipped 49 previous similar messages [ 3060.116331] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3060.825754] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 3060.825792] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 3070.623518] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3072.701144] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3083.788516] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:43:42 (1776181422) [ 3092.189891] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3117.736700] LDISKFS-fs (dm-0): recovery complete [ 3117.739537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.479528] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3127.333648] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 3127.333693] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 3133.337606] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3134.343408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3141.118351] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:44:40 (1776181480) [ 3148.394436] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3175.061963] LDISKFS-fs (dm-0): recovery complete [ 3175.065024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.647895] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 3180.846690] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3189.780885] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3191.327764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3199.251145] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:45:37 (1776181537) [ 3200.317803] Lustre: 102306:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c3202855-adee-42cc-8c9b-dd19f330dea1 at adminstrative request [ 3210.222990] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:45:49 (1776181549) [ 3215.185723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3227.595981] LDISKFS-fs (dm-0): recovery complete [ 3227.603607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.945524] LustreError: 103588:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3227.955309] LustreError: 103588:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3227.968686] Lustre: 103621:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3227.979105] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3227.987814] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3228.003817] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3228.076111] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 3228.079035] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1517 to 0x0:1537 [ 3233.249146] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3233.673351] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3253.807265] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 11:46:32 (1776181592) [ 3260.401267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3276.432338] LDISKFS-fs (dm-0): recovery complete [ 3276.439281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.725582] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3277.788240] LustreError: 105507:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3277.794418] LustreError: 105507:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3277.812766] Lustre: 105540:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3277.823401] Lustre: 105540:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3277.838239] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3277.847581] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3277.871765] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3277.957699] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1479 to 0x0:1505 [ 3277.962266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1543 to 0x0:1569 [ 3282.018336] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3282.930257] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3296.695321] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3296.700516] Lustre: Skipped 1 previous similar message [ 3302.434319] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:47:21 (1776181641) [ 3308.675831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3310.737424] Lustre: Failing over lustre-MDT0000 [ 3310.747292] Lustre: Skipped 10 previous similar messages [ 3311.069140] Lustre: server umount lustre-MDT0000 complete [ 3311.072504] Lustre: Skipped 10 previous similar messages [ 3313.633083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.647901] Lustre: Skipped 40 previous similar messages [ 3321.083277] LDISKFS-fs (dm-0): recovery complete [ 3321.086276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.208101] LustreError: 107425:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3326.211922] LustreError: 107425:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3326.216075] Lustre: 107459:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3326.226378] Lustre: 107459:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3326.234980] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3326.241778] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3326.266582] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3326.304105] Lustre: lustre-MDT0000: Denying connection for new client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 2 evicted) already passed deadline 55:26 [ 3326.335847] Lustre: Skipped 19 previous similar messages [ 3326.373291] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1512 to 0x0:1537 [ 3326.374120] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1601 [ 3330.667906] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3331.566041] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3351.057938] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:48:09 (1776181689) [ 3352.123677] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3352.125672] LustreError: 7341:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000008166836f x1862458041524608/t201863462916(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:442/0 lens 504/456 e 0 to 0 dl 1776181732 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 3357.025103] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.042905] LustreError: Skipped 363 previous similar messages [ 3364.284462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.465667] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af1dbcb to 0x2b81c8ee0af1e016 [ 3370.472423] Lustre: Skipped 10 previous similar messages [ 3370.773410] LustreError: 108804:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3370.778320] LustreError: 108804:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3370.786570] Lustre: 108838:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3370.795089] Lustre: 108838:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3370.803244] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3370.815710] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3370.841963] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3370.894997] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1633 [ 3370.898177] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1569 [ 3375.614843] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3376.108981] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3395.445961] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 11:48:53 (1776181733) [ 3403.151189] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3413.986518] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181747/real 1776181747] req@000000002ba4d2f7 x1862458061819584/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181754 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3414.007586] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3414.017672] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3414.025854] LustreError: Skipped 10 previous similar messages [ 3427.336692] LDISKFS-fs (dm-0): recovery complete [ 3427.339796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.932873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3431.935937] Lustre: Skipped 7 previous similar messages [ 3434.332524] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3437.140931] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 3437.174944] Lustre: Skipped 7 previous similar messages [ 3437.250083] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1601 [ 3437.251695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1665 [ 3445.893416] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:49:43 (1776181783) [ 3457.138908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3473.899861] LDISKFS-fs (dm-0): recovery complete [ 3473.902968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.117462] LustreError: 112571:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3475.123381] LustreError: 112571:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3475.129587] Lustre: 112604:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3475.136282] Lustre: 112604:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3475.142404] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3475.149894] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3475.175845] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3475.246143] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1633 [ 3475.261468] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1697 [ 3479.239448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3480.545448] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3501.412913] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:50:39 (1776181839) [ 3525.012889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3526.626350] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3526.633644] LustreError: Skipped 9 previous similar messages [ 3526.640553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.655545] Lustre: Skipped 1 previous similar message [ 3548.122271] LDISKFS-fs (dm-0): recovery complete [ 3548.129546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.059577] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3555.063954] Lustre: Skipped 20 previous similar messages [ 3558.695787] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3560.476450] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113 [ 3560.476852] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049 [ 3565.113111] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3566.373564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3593.608196] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:52:12 (1776181932) [ 3613.011771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3645.143425] LDISKFS-fs (dm-0): recovery complete [ 3645.147040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.329843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.344502] Lustre: Skipped 10 previous similar messages [ 3650.643647] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3654.280547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 3654.349221] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465 [ 3659.108819] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3660.430437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3677.278303] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:53:36 (1776182016) [ 3678.634565] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3680.135063] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:53:39 (1776182019) [ 3682.157497] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3683.012995] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3683.017464] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3683.030136] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to (at 0@lo) [ 3683.032484] Lustre: Skipped 54 previous similar messages [ 3683.034233] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2497 [ 3688.752597] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:53:47 (1776182027) [ 3705.761255] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3712.999687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.010899] Lustre: Skipped 2 previous similar messages [ 3738.342663] LDISKFS-fs (dm-2): recovery complete [ 3738.348757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.645255] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3740.660067] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977 [ 3742.524997] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3797.190116] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:55:36 (1776182136) [ 3806.921157] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3830.893042] LDISKFS-fs (dm-0): recovery complete [ 3830.894984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.582773] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3839.529616] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3839.534595] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009 [ 3845.275450] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3846.626967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3865.448243] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:56:44 (1776182204) [ 3870.979831] LustreError: 22628:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3876.319158] LustreError: 22628:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3876.332322] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3876.424542] LustreError: 44559:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3878.051638] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3883.487133] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3883.490664] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3883.999439] LustreError: 122376:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3884.006923] LustreError: 8671:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3884.015564] LustreError: 8671:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3885.028873] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2913 [ 3885.432069] LustreError: 7341:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3890.657362] LustreError: 7341:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3890.673147] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3890.696055] Lustre: Skipped 1 previous similar message [ 3891.880965] LustreError: 22628:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3897.312917] LustreError: 22628:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3897.316900] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3899.410453] LustreError: 22628:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3904.479203] LustreError: 22628:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3911.647247] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3911.661231] Lustre: Skipped 1 previous similar message [ 3913.495394] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3913.499586] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3918.815416] LustreError: 6261:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3918.824854] LustreError: 6261:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3932.640249] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3932.651555] Lustre: Skipped 2 previous similar messages [ 3934.355559] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3934.362806] LustreError: 6262:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3939.808439] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3939.811628] LustreError: 6262:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3948.186831] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:58:06 (1776182286) [ 3950.322320] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3956.594937] Lustre: lustre-MDT0000: Export 00000000e1186e3b already connecting from 192.168.202.32@tcp [ 3958.206593] Lustre: lustre-MDT0000: Export 00000000e1186e3b already connecting from 192.168.202.32@tcp [ 3959.901068] Lustre: lustre-MDT0000: Export 00000000e1186e3b already connecting from 192.168.202.32@tcp [ 3963.013314] Lustre: lustre-MDT0000: Export 00000000e1186e3b already connecting from 192.168.202.32@tcp [ 3963.018018] Lustre: Skipped 1 previous similar message [ 3967.458518] Lustre: lustre-MDT0000: Export 00000000e1186e3b already connecting from 192.168.202.32@tcp [ 3967.465571] Lustre: Skipped 3 previous similar messages [ 3972.688310] LustreError: 6261:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3972.695386] Lustre: 6261:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@0000000089d250d7 x1862458042555392/t0(0) o38->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:0/0 lens 520/416 e 0 to 0 dl 1776182310 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3977.060300] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 3977.083273] Lustre: Skipped 2 previous similar messages [ 3977.888808] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:58:36 (1776182316) [ 3984.202285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3986.681490] Lustre: Failing over lustre-MDT0000 [ 3986.685407] Lustre: Skipped 7 previous similar messages [ 3986.872885] Lustre: server umount lustre-MDT0000 complete [ 3986.876219] Lustre: Skipped 7 previous similar messages [ 3987.307696] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.320870] LustreError: Skipped 209 previous similar messages [ 3987.936403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.949491] Lustre: Skipped 31 previous similar messages [ 3998.053885] LDISKFS-fs (dm-0): recovery complete [ 3998.059036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.201933] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af5038e to 0x2b81c8ee0af50d35 [ 3998.209458] Lustre: Skipped 5 previous similar messages [ 3998.339349] Lustre: *** cfs_fail_loc=712, val=0*** [ 3998.341320] LustreError: 42501:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@000000009465a3c9 x1862458062177856/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3998.367501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3998.453410] LustreError: 127047:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3998.458811] LustreError: 127047:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3998.461440] Lustre: 127081:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3998.465466] Lustre: 127081:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3998.470483] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3998.476314] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3998.506231] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3998.561324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041 [ 3998.569245] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2945 [ 4002.990743] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4003.814328] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4025.823286] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182359/real 1776182359] req@0000000073da9b19 x1862458062186304/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182366 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 4025.838163] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4025.849053] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4025.870530] LustreError: Skipped 5 previous similar messages [ 4037.464203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.621149] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4043.628731] Lustre: Skipped 4 previous similar messages [ 4047.640655] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4048.894495] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4048.898102] Lustre: Skipped 4 previous similar messages [ 4048.932789] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073 [ 4048.932993] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2977 [ 4054.701682] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4055.714833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4062.128123] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:00:01 (1776182401) [ 4062.322359] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 4070.063967] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:00:09 (1776182409) [ 4071.178094] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4071.186257] LustreError: 109650:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007f2e42a8 x1862458042581056/t0(0) o700->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:372/0 lens 264/248 e 0 to 0 dl 1776182417 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 4100.819865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.015403] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4113.983507] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2979 to 0x0:3009 [ 4113.984446] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105 [ 4120.483221] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4121.856075] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4129.487678] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:01:08 (1776182468) [ 4149.389730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.641728] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 4151.649149] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137 [ 4154.252766] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4164.559797] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4165.954921] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4237.911725] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:02:56 (1776182576) [ 4247.720751] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4252.139192] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4252.160142] LustreError: Skipped 9 previous similar messages [ 4272.648389] LDISKFS-fs (dm-0): recovery complete [ 4272.651405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.975488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.981362] Lustre: Skipped 6 previous similar messages [ 4277.014520] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4277.022699] Lustre: Skipped 9 previous similar messages [ 4281.492946] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4282.905458] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 4282.909818] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3030 to 0x0:3073 [ 4354.384273] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:04:52 (1776182692) [ 4356.202348] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4356.213865] Lustre: Skipped 2 previous similar messages [ 4356.218805] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4356.221941] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201 [ 4356.223030] Lustre: Skipped 31 previous similar messages [ 4357.098293] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233 [ 4368.910849] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:05:07 (1776182707) [ 4392.981778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.570295] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4398.572454] LustreError: 135212:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000026b26595 x1862458042635904/t0(0) o101->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:731/0 lens 328/344 e 0 to 0 dl 1776182776 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4398.881569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4439.383696] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 4439.484905] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3085 to 0x0:3105 [ 4439.487297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3265 [ 4447.118766] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4449.090836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4459.249810] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:06:37 (1776182797) [ 4461.750465] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4472.089718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4499.445296] LDISKFS-fs (dm-0): recovery complete [ 4499.447918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.765820] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4521.038629] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137 [ 4521.041359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3297 [ 4531.621468] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4533.659214] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4542.244508] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:08:00 (1776182880) [ 4543.378491] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4552.499283] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4579.375393] LDISKFS-fs (dm-0): recovery complete [ 4579.378810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.452720] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4587.065687] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3169 [ 4587.067910] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329 [ 4596.303437] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4597.925642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4606.578851] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:09:05 (1776182945) [ 4607.642285] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4616.571857] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 4616.586447] Lustre: Skipped 2 previous similar messages [ 4617.429902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4619.403683] Lustre: Failing over lustre-MDT0000 [ 4619.406921] Lustre: Skipped 7 previous similar messages [ 4619.520541] Lustre: server umount lustre-MDT0000 complete [ 4619.522888] Lustre: Skipped 7 previous similar messages [ 4621.681466] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.692495] LustreError: Skipped 258 previous similar messages [ 4622.817451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.831470] Lustre: Skipped 34 previous similar messages [ 4629.983169] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182963/real 1776182963] req@000000001398d709 x1862458062350080/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182970 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 4630.003389] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4630.012259] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4630.024491] LustreError: Skipped 5 previous similar messages [ 4642.635709] LDISKFS-fs (dm-0): recovery complete [ 4642.638693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.400062] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af544c5 to 0x2b81c8ee0af54b7f [ 4647.405617] Lustre: Skipped 6 previous similar messages [ 4648.803350] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4648.816245] Lustre: Skipped 6 previous similar messages [ 4652.736887] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4653.097642] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4653.102380] Lustre: Skipped 6 previous similar messages [ 4653.154663] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201 [ 4653.155603] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3361 [ 4665.568932] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:10:04 (1776183004) [ 4667.922178] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4667.928560] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4667.934597] LustreError: 6265:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000009ec3f1d8 x1862458042668032/t261993005072(0) o35->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:214/0 lens 392/456 e 0 to 0 dl 1776183014 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4690.951895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.276830] Lustre: 6265:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000007796abf4 x1862458042668032/t261993005072(0) o35->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:249/0 lens 392/456 e 0 to 0 dl 1776183049 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4703.305358] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233 [ 4703.315817] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3393 [ 4703.910621] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4715.481850] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4717.492323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4726.632973] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:11:04 (1776183064) [ 4727.629922] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4727.634811] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000f0b88378 x1862458042676736/t266287972368(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:307/0 lens 504/448 e 0 to 0 dl 1776183107 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4736.276889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4761.355366] LDISKFS-fs (dm-0): recovery complete [ 4761.358448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.570755] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4768.777525] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425 [ 4768.787465] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3265 [ 4768.805461] Lustre: 6261:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000009a41de51 x1862458042676736/t266287972368(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:348/0 lens 504/448 e 0 to 0 dl 1776183148 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4777.023918] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4778.709255] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4788.533837] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:12:07 (1776183127) [ 4789.669523] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4789.672533] LustreError: 6263:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000622f8571 x1862458042685440/t270582939664(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:369/0 lens 504/448 e 0 to 0 dl 1776183169 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4791.495984] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4798.140137] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4798.822703] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000dc552c2c x1862458042685440/t270582939664(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:379/0 lens 504/448 e 0 to 0 dl 1776183179 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4798.856874] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4824.117669] LDISKFS-fs (dm-0): recovery complete [ 4824.124760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.787208] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4833.876416] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 4833.878290] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3457 [ 4844.078888] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:13:02 (1776183182) [ 4845.073150] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4845.078675] Lustre: Skipped 1 previous similar message [ 4845.084702] LustreError: 41251:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000dccb5eea x1862458042693440/t274877906960(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:425/0 lens 504/448 e 0 to 0 dl 1776183225 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4845.104750] LustreError: 41251:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4846.906329] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4846.908015] Lustre: Skipped 1 previous similar message [ 4853.536835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4854.132557] Lustre: 6261:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000003652631a x1862458042693440/t274877906960(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:434/0 lens 504/448 e 0 to 0 dl 1776183234 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4859.360447] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4859.371631] LustreError: Skipped 6 previous similar messages [ 4881.951962] LDISKFS-fs (dm-0): recovery complete [ 4881.954439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.131246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.133352] Lustre: Skipped 7 previous similar messages [ 4883.168275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4883.178212] Lustre: Skipped 7 previous similar messages [ 4887.827419] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4888.755092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489 [ 4888.760235] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3329 [ 4900.058978] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:13:58 (1776183238) [ 4901.168066] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4903.086898] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4903.089434] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4903.096949] LustreError: 6265:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000e9059609 x1862458042701120/t279172874256(0) o35->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:449/0 lens 392/456 e 0 to 0 dl 1776183249 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4910.456679] Lustre: 6265:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000b0db8012 x1862458042701120/t279172874256(0) o35->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:456/0 lens 392/456 e 0 to 0 dl 1776183256 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4911.276880] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4936.417408] LDISKFS-fs (dm-0): recovery complete [ 4936.426460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.972558] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4944.473361] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3521 [ 4944.474021] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 4953.962410] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:14:52 (1776183292) [ 4954.697668] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4954.707869] LustreError: 6262:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000c8eb8f91 x1862458042707520/t283467841549(0) o101->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:534/0 lens 664/600 e 0 to 0 dl 1776183334 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4996.450760] Lustre: 7341:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000003a31b98f x1862458042707520/t283467841549(0) o101->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:576/0 lens 664/3424 e 0 to 0 dl 1776183376 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 5001.921797] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:15:40 (1776183340) [ 5007.735919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5031.399199] LDISKFS-fs (dm-0): recovery complete [ 5031.402018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.491639] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 5035.496279] Lustre: Skipped 46 previous similar messages [ 5039.615439] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5041.230429] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 5041.233060] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3553 [ 5049.780710] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5051.593412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5069.429993] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:16:48 (1776183408) [ 5077.044938] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5102.114722] LDISKFS-fs (dm-0): recovery complete [ 5102.117742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.271456] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5111.877845] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3425 [ 5111.878097] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3555 to 0x0:3585 [ 5120.501570] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5121.904645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5126.792818] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5137.443695] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:17:56 (1776183476) [ 5176.759661] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5201.851031] LDISKFS-fs (dm-0): recovery complete [ 5201.854613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.625786] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5214.044135] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4705 [ 5214.045544] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865 [ 5222.461299] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5224.166068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5287.621215] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:20:26 (1776183626) [ 5296.351300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5297.905822] Lustre: Failing over lustre-MDT0000 [ 5297.907710] Lustre: Skipped 8 previous similar messages [ 5298.086412] Lustre: server umount lustre-MDT0000 complete [ 5298.089351] Lustre: Skipped 8 previous similar messages [ 5299.564575] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5299.574428] LustreError: Skipped 343 previous similar messages [ 5300.703924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.708593] Lustre: Skipped 34 previous similar messages [ 5306.847183] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183640/real 1776183640] req@00000000bebde186 x1862458062862976/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183647 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 5306.879911] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 5306.889668] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.900208] LustreError: Skipped 8 previous similar messages [ 5319.249532] LDISKFS-fs (dm-0): recovery complete [ 5319.253654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.284938] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0af6d23d to 0x2b81c8ee0af983bd [ 5324.295719] Lustre: Skipped 8 previous similar messages [ 5325.185087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5325.194599] Lustre: Skipped 8 previous similar messages [ 5329.089856] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5330.048036] Lustre: lustre-MDT0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 5330.058869] Lustre: Skipped 8 previous similar messages [ 5330.110090] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4897 [ 5330.113122] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4707 to 0x0:4737 [ 5338.656725] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5340.224040] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5351.775084] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5353.432764] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5358.944317] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:21:37 (1776183697) [ 5366.387668] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5410.142057] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 5410.153861] Lustre: Skipped 4 previous similar messages [ 5413.299072] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5413.301108] Lustre: Skipped 1 previous similar message [ 5413.314326] LustreError: 22628:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000001637fe7b x1862458044195648/t300647710728(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:238/0 lens 66040/440 e 0 to 0 dl 1776183793 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5455.209351] Lustre: 22628:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000006931aea5 x1862458044195648/t300647710728(0) o36->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:280/0 lens 66040/440 e 0 to 0 dl 1776183835 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5465.936803] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5467.740736] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:23:26 (1776183806) [ 5480.142386] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5503.525084] LDISKFS-fs (dm-0): recovery complete [ 5503.533435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.231619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.235416] Lustre: Skipped 5 previous similar messages [ 5507.267913] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5507.276277] Lustre: Skipped 5 previous similar messages [ 5511.546296] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5513.679921] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4999 to 0x0:5025 [ 5513.680959] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4838 to 0x0:4865 [ 5520.081729] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5521.255614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5528.827315] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:24:27 (1776183867) [ 5546.803600] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5577.866931] LDISKFS-fs (dm-2): recovery complete [ 5577.874583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5582.734771] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5596.797894] LustreError: 166330:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5596.807514] Lustre: 165810:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5596.816055] Lustre: 165810:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 5596.824592] Lustre: 165810:0:(ldlm_lib.c:1804:abort_req_replay_queue()) @@@ aborted: req@0000000019ae1dc3 x1862458062952192/t0(17179870572) o6->lustre-MDT0000-mdtlov_UUID@0@lo:393/0 lens 544/0 e 2 to 0 dl 1776183948 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'osp-syn-0-0.0' [ 5596.838219] LustreError: 165810:0:(ofd_obd.c:1309:ofd_iocontrol()) lustre-OST0000: not supported cmd = 0xc00866c1 [ 5596.838520] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -19 [ 5596.848899] LustreError: Skipped 3 previous similar messages [ 5596.851947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.853223] Lustre: 165810:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5596.855441] Lustre: Skipped 1 previous similar message [ 5616.383749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5620.335701] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5623.605475] LustreError: 3357:0:(client.c:3161:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@00000000c21bfacb x1862458062952192/t17179870572(17179870572) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 3 to 0 dl 1776183976 ref 2 fl Interpret:RQU/4/0 rc 0/0 job:'osp-syn-0-0.0' [ 5624.258760] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5624.259233] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 5629.579321] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5631.346787] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5671.027659] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:26:49 (1776184009) [ 5692.914634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.557430] Lustre: MGC192.168.202.132@tcp: Connection restored to 192.168.202.132@tcp (at 0@lo) [ 5699.562218] Lustre: Skipped 26 previous similar messages [ 5704.676289] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5705.261372] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473 [ 5705.262423] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5281 [ 5737.340898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.664674] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5749.863868] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505 [ 5749.868830] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5313 [ 5760.024118] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.565380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5770.651187] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:28:29 (1776184109) [ 5803.782247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5806.116379] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5806.120576] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5537 [ 5809.015870] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5817.367631] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5818.719713] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5827.804179] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:29:26 (1776184166) [ 5840.285573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.412428] Lustre: *** cfs_fail_loc=605, val=0*** [ 5840.418390] LustreError: 172393:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=0000000076b8c830 failed: rc = -95 [ 5840.446524] LustreError: 172393:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5840.456635] LustreError: 172393:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5840.459418] LustreError: 172393:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5840.471176] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5840.482346] LustreError: 172393:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5840.510466] LustreError: 172393:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5847.325546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.942968] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5315 to 0x0:5345 [ 5863.948431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5569 [ 5863.974710] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5873.581555] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:30:12 (1776184212) [ 5882.729364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5900.608706] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.631611] LustreError: Skipped 234 previous similar messages [ 5916.254793] LDISKFS-fs (dm-0): recovery complete [ 5916.274623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.447604] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0afacd50 to 0x2b81c8ee0afad55a [ 5929.467396] Lustre: Skipped 4 previous similar messages [ 5930.337027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5930.354625] Lustre: Skipped 7 previous similar messages [ 5930.370914] Lustre: *** cfs_fail_loc=707, val=0*** [ 5934.658606] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5971.353390] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 5971.892804] Lustre: lustre-MDT0000: Recovery over after 0:41, of 2 clients 2 recovered and 0 were evicted. [ 5971.907698] Lustre: Skipped 7 previous similar messages [ 5971.962937] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5358 to 0x0:5377 [ 5971.963731] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5583 to 0x0:5601 [ 5979.532206] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5981.515733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5991.497257] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:32:10 (1776184330) [ 6022.618796] LustreError: 41251:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 6028.671133] LustreError: 41251:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6047.530645] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:33:06 (1776184386) [ 6078.290168] LustreError: 40918:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 6084.335946] LustreError: 40918:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 6094.550638] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:33:53 (1776184433) [ 6125.019971] LustreError: 6262:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6130.023065] LustreError: 6262:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6132.146674] LustreError: 6262:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 6142.159789] LustreError: 6262:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6160.045351] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:34:58 (1776184498) [ 6217.452696] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:35:56 (1776184556) [ 6247.473661] LustreError: 6262:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6247.903472] LustreError: 6262:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6255.650546] LustreError: 6264:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6255.669397] LustreError: 6264:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 6256.104891] LustreError: 6264:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6256.109787] LustreError: 6264:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 6271.979712] LustreError: 22628:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6271.984988] LustreError: 22628:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6272.407187] LustreError: 22628:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6272.411310] LustreError: 22628:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6274.768388] LustreError: 14513:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 6282.439231] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:37:00 (1776184620) [ 6318.785422] Lustre: DEBUG MARKER: phase 2 [ 6329.553661] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:37:47 (1776184667) [ 6417.063044] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:39:15 (1776184755) [ 6418.678694] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6420.571395] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:39:19 (1776184759) [ 6425.538614] Lustre: DEBUG MARKER: Started rundbench load pid=138656 ... [ 6434.565755] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6437.297340] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6439.264781] Lustre: Failing over lustre-MDT0000 [ 6439.279285] Lustre: Skipped 8 previous similar messages [ 6439.363105] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 6439.551063] Lustre: server umount lustre-MDT0000 complete [ 6439.554049] Lustre: Skipped 9 previous similar messages [ 6439.908752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.916489] Lustre: Skipped 25 previous similar messages [ 6452.191168] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184785/real 1776184785] req@00000000cc141175 x1862458063256640/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184792 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 6452.217550] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 6452.226452] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6452.238011] LustreError: Skipped 5 previous similar messages [ 6463.562494] LDISKFS-fs (dm-0): recovery complete [ 6463.572079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.613107] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 6469.633328] Lustre: Skipped 21 previous similar messages [ 6470.039576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6470.047199] Lustre: Skipped 7 previous similar messages [ 6470.083328] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6470.096795] Lustre: Skipped 7 previous similar messages [ 6476.752257] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6477.689922] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5703 to 0x0:5729 [ 6477.693319] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5460 to 0x0:5505 [ 6491.279819] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6493.757180] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6505.440094] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6508.247784] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6511.079101] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6511.084051] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.083264] LustreError: Skipped 2 previous similar messages [ 6511.094621] LustreError: Skipped 84 previous similar messages [ 6536.691023] LDISKFS-fs (dm-1): recovery complete [ 6536.695098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6538.266347] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6538.286190] Lustre: Skipped 1 previous similar message [ 6542.404535] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:166 to 0x280000400:193 [ 6542.405377] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 [ 6542.407241] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6553.971663] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6556.102792] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6577.473379] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:41:55 (1776184915) [ 6708.816257] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6722.457703] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6724.809599] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [ 6750.173558] LDISKFS-fs (dm-1): recovery complete [ 6750.179175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.968054] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6760.750094] Lustre: lustre-MDT0001: Recovery over after 0:09, of 2 clients 2 recovered and 0 were evicted. [ 6760.762716] Lustre: Skipped 2 previous similar messages [ 6760.888790] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:325 to 0x2c0000400:353 [ 6760.890477] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:486 to 0x280000400:513 [ 6766.800752] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6768.484965] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6820.280621] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:45:57 (1776185157) [ 6951.485479] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6963.689613] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6965.312185] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 6990.747250] LDISKFS-fs (dm-0): recovery complete [ 6990.752217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.429597] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0afb3e37 to 0x2b81c8ee0b0559c8 [ 6995.465327] Lustre: Skipped 1 previous similar message [ 7000.965503] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7003.336421] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6276 to 0x0:6305 [ 7003.340668] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6053 to 0x0:6081 [ 7011.500347] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7013.451549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7023.394437] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:49:21 (1776185361) [ 7152.955912] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7165.557956] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7167.338996] Lustre: Failing over lustre-MDT0001 [ 7167.342147] Lustre: Skipped 3 previous similar messages [ 7167.486191] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7167.493963] LustreError: Skipped 2 previous similar messages [ 7167.496347] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7167.501595] Lustre: Skipped 10 previous similar messages [ 7167.503860] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.509914] LustreError: Skipped 84 previous similar messages [ 7167.533273] Lustre: server umount lustre-MDT0001 complete [ 7167.534708] Lustre: Skipped 3 previous similar messages [ 7190.634132] LDISKFS-fs (dm-1): recovery complete [ 7190.637349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.865540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7190.872521] Lustre: Skipped 3 previous similar messages [ 7190.891539] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7190.899323] Lustre: Skipped 3 previous similar messages [ 7191.947415] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7191.954273] Lustre: Skipped 2 previous similar messages [ 7194.982796] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7196.143112] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 7196.156304] Lustre: Skipped 15 previous similar messages [ 7196.236649] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:493 to 0x2c0000400:513 [ 7196.237515] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:654 to 0x280000400:673 [ 7203.990702] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7205.354675] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7212.351262] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:52:31 (1776185551) [ 7223.768428] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7225.183445] Lustre: lustre-OST0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnecting [ 7225.195260] Lustre: Skipped 1 previous similar message [ 7226.693350] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7252.742767] LDISKFS-fs (dm-2): recovery complete [ 7252.749658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7254.464085] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:654 to 0x280000400:705 [ 7254.468821] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6921 to 0x0:6945 [ 7258.153685] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7268.003108] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7269.653361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7279.749429] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:53:38 (1776185618) [ 7410.401722] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7418.923481] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7431.220523] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7433.313036] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 7437.177081] LustreError: 6245:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776185777 with bad export cookie 3135007740339247560 [ 7437.181769] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7437.184922] LustreError: 6245:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7437.200140] LustreError: Skipped 1 previous similar message [ 7463.062786] LDISKFS-fs (dm-0): recovery complete [ 7463.070966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.196210] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7505.631444] LDISKFS-fs (dm-1): recovery complete [ 7505.636169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.944141] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7511.566641] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7511.573844] Lustre: Skipped 3 previous similar messages [ 7511.595613] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:493 to 0x2c0000400:545 [ 7511.596553] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:654 to 0x280000400:737 [ 7518.638884] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6921 to 0x0:6977 [ 7518.643341] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6697 to 0x0:6721 [ 7525.223198] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7526.911260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7528.214309] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7536.682338] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 12:57:55 (1776185875) [ 7544.144735] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7554.527169] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185887/real 1776185887] req@000000006f991a35 x1862458066551488/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185894 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 7554.543789] Lustre: 3360:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7569.724224] LDISKFS-fs (dm-0): recovery complete [ 7569.731978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.826662] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7576.406043] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7579.041081] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 7579.086962] Lustre: 195588:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ca9cde77 x1862458050730752/t330712488317(330712488317) o101->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:105/0 lens 648/3424 e 0 to 0 dl 1776185925 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7579.192121] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6753 [ 7579.192283] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6921 to 0x0:7009 [ 7586.208559] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7587.866302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7597.002913] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 12:58:55 (1776185935) [ 7604.724259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7607.271532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7607.279678] Lustre: Skipped 5 previous similar messages [ 7629.681570] LDISKFS-fs (dm-0): recovery complete [ 7629.685639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.479293] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0b0e3cd6 to 0x2b81c8ee0b0e4232 [ 7637.502319] Lustre: Skipped 2 previous similar messages [ 7639.800871] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7639.804664] LustreError: 196118:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000004227941e x1862458050730752/t330712488317(330712488317) o101->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:166/0 lens 648/600 e 0 to 0 dl 1776185986 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7641.562697] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7646.052519] Lustre: lustre-MDT0000: Client c3202855-adee-42cc-8c9b-dd19f330dea1 (at 192.168.202.32@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7646.064395] Lustre: 197371:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000c8eb8f91 x1862458050730752/t330712488317(330712488317) o101->c3202855-adee-42cc-8c9b-dd19f330dea1@192.168.202.32@tcp:172/0 lens 648/3424 e 0 to 0 dl 1776185992 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7646.176388] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6785 [ 7646.178307] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6921 to 0x0:7041 [ 7650.785204] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7652.376369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7660.248185] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 12:59:59 (1776185999) [ 7686.160749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.966555] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6817 [ 7698.181980] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7709.201278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.969682] Lustre: lustre-OST0000: Denying connection for new client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7711.740832] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6921 to 0x0:7073 [ 7711.740903] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:654 to 0x280000400:769 [ 7714.813973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7726.298827] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:01:05 (1776186065) [ 7727.603301] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7727.610643] LustreError: 185215:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000001060e90 x1862458066606400/t347892350983(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:253/0 lens 1056/4320 e 0 to 0 dl 1776186073 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7734.693632] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7734.763897] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7763.985207] LDISKFS-fs (dm-0): recovery complete [ 7763.993352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7769.668334] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7770.223296] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7105 [ 7770.225205] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6849 [ 7778.341198] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7779.959505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7788.758363] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:02:07 (1776186127) [ 7789.882709] LustreError: 185215:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000065359a94 x1862458066627264/t352187318282(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:316/0 lens 2200/4320 e 0 to 0 dl 1776186136 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7797.224389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7797.253429] Lustre: Skipped 27 previous similar messages [ 7797.278937] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7797.305291] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 7797.332532] Lustre: Skipped 33 previous similar messages [ 7798.944347] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7810.297841] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7813.107292] Lustre: Failing over lustre-MDT0001 [ 7813.114708] Lustre: Skipped 8 previous similar messages [ 7813.274123] Lustre: server umount lustre-MDT0001 complete [ 7813.281721] Lustre: Skipped 8 previous similar messages [ 7813.532932] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.576056] LustreError: Skipped 220 previous similar messages [ 7816.167974] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7816.186258] LustreError: Skipped 5 previous similar messages [ 7840.501498] LDISKFS-fs (dm-1): recovery complete [ 7840.508506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.883256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7840.888402] Lustre: Skipped 8 previous similar messages [ 7840.921962] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7840.931031] Lustre: Skipped 8 previous similar messages [ 7842.523938] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7842.537752] Lustre: Skipped 8 previous similar messages [ 7846.189541] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7846.476660] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:556 to 0x2c0000400:577 [ 7846.477990] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:780 to 0x280000400:801 [ 7856.943139] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7858.637569] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7867.222480] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:03:26 (1776186206) [ 7868.366533] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7868.370923] Lustre: Skipped 1 previous similar message [ 7868.374906] LustreError: 195593:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000a410289b x1862458066652352/t352187318301(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:394/0 lens 2200/4320 e 0 to 0 dl 1776186214 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7875.554114] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7876.118536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7883.799854] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7912.947601] LDISKFS-fs (dm-0): recovery complete [ 7912.951729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7920.645325] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7921.334878] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6881 [ 7921.335831] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7137 [ 7928.613662] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7929.975822] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7958.583316] LDISKFS-fs (dm-1): recovery complete [ 7958.585517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.028847] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7964.217934] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:588 to 0x2c0000400:609 [ 7964.220898] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:812 to 0x280000400:833 [ 7973.230824] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7974.386382] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7984.155539] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:05:22 (1776186322) [ 7992.288708] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7995.646910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8005.346060] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8007.657872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8012.607063] LustreError: 7257:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186352 with bad export cookie 3135007740339840075 [ 8012.619496] LustreError: 7257:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8041.885813] LDISKFS-fs (dm-0): recovery complete [ 8041.888658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8052.441398] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8075.316830] LDISKFS-fs (dm-1): recovery complete [ 8075.324769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.380787] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8081.125313] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7169 [ 8081.126328] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6913 [ 8081.235526] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:620 to 0x2c0000400:641 [ 8081.238903] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:844 to 0x280000400:865 [ 8090.237616] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8091.540304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8092.924373] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8102.734915] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:07:20 (1776186440) [ 8103.874925] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8103.879447] Lustre: Skipped 1 previous similar message [ 8103.887067] LustreError: 214108:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000eac608d6 x1862458051690688/t34359738438(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:630/0 lens 560/448 e 0 to 0 dl 1776186450 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8103.920236] LustreError: 214108:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 8110.957197] Lustre: lustre-MDT0001: Client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp) reconnecting [ 8110.995485] Lustre: 214108:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000f1c2bc5f x1862458051690688/t34359738438(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:637/0 lens 560/448 e 0 to 0 dl 1776186457 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8114.711307] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8128.991615] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8129.001276] LustreError: Skipped 6 previous similar messages [ 8140.502294] LDISKFS-fs (dm-0): recovery complete [ 8140.504464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.125788] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8152.130111] Lustre: Skipped 11 previous similar messages [ 8152.214724] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6945 [ 8152.347531] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8163.759850] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8165.379721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8174.779581] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:08:33 (1776186513) [ 8183.034191] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8183.184610] Lustre: lustre-MDT0001: Client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp) reconnecting [ 8183.210544] Lustre: 214107:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000f3051543 x1862458051708224/t34359738508(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:709/0 lens 560/448 e 0 to 0 dl 1776186529 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8208.452778] LDISKFS-fs (dm-1): recovery complete [ 8208.457893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.051154] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:886 to 0x280000400:929 [ 8214.061206] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:662 to 0x2c0000400:705 [ 8215.333511] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8225.317650] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8227.329582] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8238.049423] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:09:36 (1776186576) [ 8246.625413] Lustre: lustre-MDT0001: Client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp) reconnecting [ 8246.656768] Lustre: 214107:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000001dffbd7 x1862458051724800/t38654705736(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:17/0 lens 560/448 e 0 to 0 dl 1776186592 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8250.145824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8258.415959] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8272.351150] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186605/real 1776186605] req@00000000e03ae23d x1862458066769088/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186612 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 8272.376184] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 8284.669824] LDISKFS-fs (dm-0): recovery complete [ 8284.672651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.826657] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0b0e81f6 to 0x2b81c8ee0b0e98ee [ 8289.849492] Lustre: Skipped 5 previous similar messages [ 8295.670445] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6977 [ 8295.670876] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7201 [ 8298.102711] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8311.375592] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8313.415057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8342.384616] LDISKFS-fs (dm-1): recovery complete [ 8342.390601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.792563] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8348.208486] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:940 to 0x280000400:961 [ 8348.211323] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:737 [ 8357.178676] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8358.648472] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8368.526322] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:11:47 (1776186707) [ 8369.533467] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8369.538482] Lustre: Skipped 2 previous similar messages [ 8369.542289] LustreError: 214108:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000001d9b42e0 x1862458051747712/t42949673030(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:140/0 lens 560/448 e 0 to 0 dl 1776186715 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8369.562167] LustreError: 214108:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 2 previous similar messages [ 8376.674723] Lustre: lustre-MDT0001: Client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp) reconnecting [ 8376.694573] Lustre: 218774:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000092b6c1c5 x1862458051747712/t42949673030(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:147/0 lens 560/448 e 0 to 0 dl 1776186722 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8379.437414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8387.980727] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8394.299899] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186734 with bad export cookie 3135007740339853550 [ 8394.311267] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8406.431358] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.447579] Lustre: Skipped 37 previous similar messages [ 8418.360588] LDISKFS-fs (dm-0): recovery complete [ 8418.364991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.936581] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 8426.958509] Lustre: Skipped 36 previous similar messages [ 8427.148238] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.175237] LustreError: Skipped 237 previous similar messages [ 8432.338918] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8445.279675] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8445.288620] Lustre: Skipped 8 previous similar messages [ 8460.363596] LDISKFS-fs (dm-1): recovery complete [ 8460.375145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8460.642519] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (not set up) [ 8460.660923] Lustre: Skipped 8 previous similar messages [ 8460.746547] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8460.759643] LustreError: Skipped 8 previous similar messages [ 8460.860560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8460.868484] Lustre: Skipped 9 previous similar messages [ 8460.912159] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8460.929694] Lustre: Skipped 9 previous similar messages [ 8466.125774] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:993 [ 8466.127835] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:748 to 0x2c0000400:769 [ 8466.379801] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7233 [ 8466.389460] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7009 [ 8466.980339] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8478.314631] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8480.230477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8482.112487] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8491.615794] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:13:50 (1776186830) [ 8500.209435] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8500.964725] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8502.801459] Lustre: Failing over lustre-MDT0001 [ 8502.803952] Lustre: Skipped 10 previous similar messages [ 8502.940638] Lustre: server umount lustre-MDT0001 complete [ 8502.943051] Lustre: Skipped 10 previous similar messages [ 8526.911736] LDISKFS-fs (dm-1): recovery complete [ 8526.916892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.518098] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1025 [ 8532.522866] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:801 [ 8532.693477] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8542.426288] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8543.801971] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8551.599797] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:14:50 (1776186890) [ 8560.097226] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8560.179908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8588.993860] LDISKFS-fs (dm-0): recovery complete [ 8588.998844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.247087] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8610.386785] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7265 [ 8610.387199] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7041 [ 8620.726479] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8622.723502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8631.613374] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:16:10 (1776186970) [ 8639.975420] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8640.125573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8646.989538] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8673.487895] LDISKFS-fs (dm-0): recovery complete [ 8673.491837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.584861] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7073 [ 8680.585251] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7297 [ 8681.542576] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8692.191366] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8694.237644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8724.811852] LDISKFS-fs (dm-1): recovery complete [ 8724.826140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.451227] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8730.713808] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1057 [ 8730.716315] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:833 [ 8743.488225] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8745.321494] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8753.732663] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:18:12 (1776187092) [ 8762.342057] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8762.813818] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8772.726883] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8779.257817] LustreError: 104335:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187119 with bad export cookie 3135007740339863861 [ 8779.263534] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8779.266221] LustreError: 104335:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8779.282160] LustreError: Skipped 4 previous similar messages [ 8782.816148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8809.183244] LDISKFS-fs (dm-0): recovery complete [ 8809.191939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.440459] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8842.441355] LDISKFS-fs (dm-1): recovery complete [ 8842.451389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8847.655432] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8848.919345] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 2 recovered and 0 were evicted. [ 8848.940240] Lustre: Skipped 9 previous similar messages [ 8849.002041] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7105 [ 8849.007434] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7329 [ 8849.126192] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1089 [ 8849.128323] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:865 [ 8859.464144] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8861.712103] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8863.217126] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8872.434972] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:20:10 (1776187210) [ 8880.473845] Lustre: lustre-MDT0001: Client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp) reconnecting [ 8880.505629] Lustre: 237752:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000362aaa40 x1862458051807616/t60129542146(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:651/0 lens 496/2888 e 0 to 0 dl 1776187226 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 8883.420051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8896.479151] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187229/real 1776187229] req@0000000062519bcc x1862458066945088/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187236 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 8896.514944] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 8911.326609] LDISKFS-fs (dm-0): recovery complete [ 8911.333029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.869800] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0b0ecab2 to 0x2b81c8ee0b0ed364 [ 8912.879144] Lustre: Skipped 4 previous similar messages [ 8918.691738] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7137 [ 8918.693308] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7361 [ 8919.112490] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8928.952368] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8930.478856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8939.848522] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:21:17 (1776187277) [ 8942.520185] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8942.544058] Lustre: Skipped 5 previous similar messages [ 8942.548910] LustreError: 237751:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000002f560d0e x1862458051816064/t60129542148(0) o36->326d7a32-8006-4d8d-8723-e14ad87d4906@192.168.202.32@tcp:713/0 lens 496/456 e 0 to 0 dl 1776187288 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 8942.571332] LustreError: 237751:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 5 previous similar messages [ 8952.444196] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8983.074637] LDISKFS-fs (dm-1): recovery complete [ 8983.077749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8987.012735] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8988.724864] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:897 [ 8988.729047] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1121 [ 8996.346373] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8998.163765] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9007.635380] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:22:26 (1776187346) [ 9016.968679] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9026.392641] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9029.600124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9029.612536] Lustre: Skipped 31 previous similar messages [ 9029.617205] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9029.633890] LustreError: Skipped 244 previous similar messages [ 9055.229054] LDISKFS-fs (dm-0): recovery complete [ 9055.233823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9057.258699] Lustre: MGC192.168.202.132@tcp: Connection restored to (at 0@lo) [ 9057.271486] Lustre: Skipped 41 previous similar messages [ 9060.188940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9060.198845] Lustre: Skipped 9 previous similar messages [ 9062.503143] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9063.011882] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7393 [ 9063.020927] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7169 [ 9072.414204] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9074.020892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9078.245523] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9078.255118] LustreError: Skipped 6 previous similar messages [ 9100.503564] LDISKFS-fs (dm-1): recovery complete [ 9100.506947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.741164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9100.744315] Lustre: Skipped 9 previous similar messages [ 9100.759878] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 9100.764030] Lustre: Skipped 9 previous similar messages [ 9105.853839] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9105.949641] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1153 [ 9105.949666] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:929 [ 9116.061453] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9117.784436] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9127.321230] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:24:26 (1776187466) [ 9134.944277] Lustre: lustre-MDT0001: Client 326d7a32-8006-4d8d-8723-e14ad87d4906 (at 192.168.202.32@tcp) reconnecting [ 9134.949681] Lustre: Skipped 2 previous similar messages [ 9135.866508] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9142.569750] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9144.325958] Lustre: Failing over lustre-MDT0000 [ 9144.333348] Lustre: Skipped 9 previous similar messages [ 9144.464668] Lustre: server umount lustre-MDT0000 complete [ 9144.467319] Lustre: Skipped 9 previous similar messages [ 9148.610872] LustreError: 7257:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187488 with bad export cookie 3135007740339871337 [ 9148.632242] LustreError: 7257:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9174.204295] LDISKFS-fs (dm-0): recovery complete [ 9174.208383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.301025] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9211.063403] LDISKFS-fs (dm-1): recovery complete [ 9211.079740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.711723] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9217.844612] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1185 [ 9217.846270] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:961 [ 9218.079069] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7201 [ 9218.089706] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7075 to 0x0:7425 [ 9228.173148] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9229.750910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9231.399723] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9240.828672] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:26:19 (1776187579) [ 9242.737023] Lustre: 251629:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 326d7a32-8006-4d8d-8723-e14ad87d4906 at adminstrative request [ 9252.900597] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:26:31 (1776187591) [ 9258.978520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9258.987830] Lustre: Skipped 4 previous similar messages [ 9277.541307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.716948] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9293.559236] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7477 to 0x0:7521 [ 9293.560536] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7252 to 0x0:7297 [ 9301.016658] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9302.588652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9309.929272] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:27:28 (1776187648) [ 9341.201487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9343.327599] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7622 to 0x0:7649 [ 9343.329716] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1217 [ 9345.336344] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9355.247455] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9356.808576] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9365.999419] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:28:24 (1776187704) [ 9379.889145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.083269] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9380.097302] LustreError: Skipped 4 previous similar messages [ 9385.497792] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7622 to 0x0:7681 [ 9385.501152] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7252 to 0x0:7329 [ 9385.704959] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9394.883624] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:28:53 (1776187733) [ 9402.263361] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9427.277390] LDISKFS-fs (dm-2): recovery complete [ 9427.287891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9429.579945] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1249 [ 9429.580889] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7683 to 0x0:7713 [ 9432.081058] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9440.568550] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9442.247465] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9449.900403] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:29:48 (1776187788) [ 9456.816299] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9483.670771] LDISKFS-fs (dm-2): recovery complete [ 9483.677086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9485.863294] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.32@tcp inode [0x20002ea31:0x5:0x0] object 0x0:7714 extent [0-4194303]: client csum b235026a, server csum fa4d02f3 [ 9486.088940] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9486.091460] Lustre: Skipped 11 previous similar messages [ 9486.097787] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7715 to 0x0:7745 [ 9486.099829] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1281 [ 9489.704665] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9500.677940] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9502.539248] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9510.725302] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:30:49 (1776187849) [ 9520.569911] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9528.661911] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9538.329133] LustreError: 44569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187878 with bad export cookie 3135007740339898049 [ 9538.338341] LustreError: 44569:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9560.564857] LDISKFS-fs (dm-0): recovery complete [ 9560.572128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.298614] Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x2b81c8ee0b0f46c1 to 0x2b81c8ee0b0f58cd [ 9571.307809] Lustre: Skipped 4 previous similar messages [ 9575.119748] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9577.386993] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7252 to 0x0:7361 [ 9596.675667] LDISKFS-fs (dm-2): recovery complete [ 9596.679026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9598.693938] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1313 [ 9601.121989] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9615.850927] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:32:34 (1776187954) [ 9632.740065] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9632.740837] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9632.752579] Lustre: Skipped 34 previous similar messages [ 9632.780658] LustreError: Skipped 260 previous similar messages [ 9638.879808] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187972/real 1776187972] req@000000002fa946a8 x1862458067157440/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187979 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 9638.910740] Lustre: 3361:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 9647.301774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.782326] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9661.943523] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 9661.950488] Lustre: Skipped 37 previous similar messages [ 9662.029401] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7252 to 0x0:7393 [ 9667.923765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9669.899639] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9669.910447] Lustre: Skipped 11 previous similar messages [ 9671.918525] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9673.814312] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 9673.831228] Lustre: Skipped 1 previous similar message [ 9679.202799] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 9684.314745] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 9689.436731] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 9694.556786] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 9704.797501] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:34 [ 9704.814875] Lustre: Skipped 1 previous similar message [ 9725.298661] Lustre: lustre-OST0000: Denying connection for new client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:13 [ 9725.329106] Lustre: Skipped 3 previous similar messages [ 9739.002486] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9739.006240] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9739.032579] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1345 [ 9739.035431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7796 to 0x0:7817 [ 9743.157504] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 60 sec [ 9752.734752] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 9760.101142] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:34:59 (1776188099) [ 9764.302288] Lustre: Failing over lustre-OST0001 [ 9764.308248] Lustre: Skipped 10 previous similar messages [ 9764.379660] Lustre: server umount lustre-OST0001 complete [ 9764.392959] Lustre: Skipped 10 previous similar messages [ 9764.840151] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9764.850184] LustreError: Skipped 9 previous similar messages [ 9785.029344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.224391] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9785.230284] Lustre: Skipped 11 previous similar messages [ 9785.244530] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 9785.247595] Lustre: Skipped 11 previous similar messages [ 9787.172394] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7396 to 0x0:7425 [ 9787.179401] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:780 to 0x2c0000400:993 [ 9790.478408] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9801.496678] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:35:40 (1776188140) [ 9823.732397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.993485] LustreError: 268295:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9825.997979] LustreError: 268295:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 9826.655448] Lustre: *** cfs_fail_loc=715, val=0*** [ 9827.680157] Lustre: *** cfs_fail_loc=715, val=0*** [ 9827.682743] Lustre: Skipped 1 previous similar message [ 9827.739168] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9831.905745] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:59 [ 9831.919145] Lustre: Skipped 1 previous similar message [ 9832.928314] Lustre: *** cfs_fail_loc=715, val=0*** [ 9833.305564] Lustre: lustre-OST0000: Client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp) reconnected, waiting for 3 clients in recovery for 0:57 [ 9839.073073] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:51 [ 9839.082991] Lustre: Skipped 1 previous similar message [ 9840.095881] Lustre: *** cfs_fail_loc=715, val=0*** [ 9840.099577] Lustre: Skipped 2 previous similar messages [ 9846.241668] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:44 [ 9846.246641] Lustre: Skipped 2 previous similar messages [ 9847.265674] Lustre: *** cfs_fail_loc=715, val=0*** [ 9847.270731] Lustre: Skipped 2 previous similar messages [ 9853.409043] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:37 [ 9853.420637] Lustre: Skipped 2 previous similar messages [ 9855.839281] Lustre: *** cfs_fail_loc=715, val=0*** [ 9855.842835] Lustre: Skipped 4 previous similar messages [ 9861.978138] Lustre: lustre-OST0000: Client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp) reconnected, waiting for 3 clients in recovery for 0:29 [ 9862.007213] Lustre: Skipped 3 previous similar messages [ 9865.999145] LustreError: 268295:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9866.019742] LustreError: 268295:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 9866.038872] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1004 to 0x280000400:1377 [ 9866.040657] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7821 to 0x0:7849 [ 9872.959523] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9874.349691] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9882.675545] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:37:01 (1776188221) [ 9903.731277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.440073] Lustre: *** cfs_fail_loc=715, val=0*** [ 9905.443699] Lustre: Skipped 3 previous similar messages [ 9908.651535] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9909.744439] LustreError: 269864:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9911.591811] Lustre: lustre-MDT0000: Client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 9915.872305] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9930.208131] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9930.216161] Lustre: Skipped 1 previous similar message [ 9938.399860] Lustre: *** cfs_fail_loc=715, val=0*** [ 9938.404049] Lustre: Skipped 8 previous similar messages [ 9947.479427] Lustre: lustre-MDT0000: Client fab95aae-13ad-4b0d-af60-125a14481788 (at 192.168.202.32@tcp) reconnected, waiting for 2 clients in recovery for 0:22 [ 9947.482916] Lustre: Skipped 4 previous similar messages [ 9951.712334] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9951.719804] Lustre: Skipped 2 previous similar messages [ 9975.139338] Lustre: lustre-MDT0000: Recovery already passed deadline 0:05. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9982.298627] Lustre: lustre-MDT0000: Recovery already passed deadline 0:12. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9986.528024] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9986.538222] Lustre: Skipped 4 previous similar messages [ 9989.461869] Lustre: lustre-MDT0000: Recovery already passed deadline 0:19. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9989.791109] LustreError: 269864:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9989.819986] Lustre: 269864:0:(ldlm_lib.c:2849:target_recovery_thread()) too long recovery - read logs [ 9989.826367] LustreError: dumping log to /tmp/lustre-log.1776188330.269864 [ 9989.910525] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7436 to 0x0:7457 [ 9989.911637] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7860 to 0x0:7881 [ 9995.179971] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9996.430352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10004.636645] Lustre: DEBUG MARKER: == replay-single test complete, duration 9731 sec ======== 13:39:03 (1776188343) [10011.636940] LustreError: 6245:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188351 with bad export cookie 3135007740339912553 [10011.654371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10019.295193] LustreError: 166-1: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10019.315294] LustreError: Skipped 3 previous similar messages [10037.698569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10057.059958] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [10057.843857] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7860 to 0x0:7913 [10057.848175] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7436 to 0x0:7489 [10068.473407] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10069.856945] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10078.177296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10078.177296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10078.188793] Lustre: Skipped 2 previous similar messages [10132.250053] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [10135.426355] Key type lgssc unregistered [10135.796720] LNet: 274707:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10135.810809] LNet: Removed LNI 192.168.202.132@tcp [10136.651300] Key type .llcrypt unregistered [10136.653327] Key type ._llcrypt unregistered