[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 642431900 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002295] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009164] LSM: Security Framework initializing [ 0.010062] Yama: becoming mindful. [ 0.011040] SELinux: Initializing. [ 0.012108] *** VALIDATE selinux *** [ 0.020519] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025847] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028181] *** VALIDATE tmpfs *** [ 0.030317] *** VALIDATE proc *** [ 0.032458] *** VALIDATE cgroup *** [ 0.033026] *** VALIDATE cgroup2 *** [ 0.035010] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036256] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038053] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042456] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.044180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045789] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046031] ... version: 2 [ 0.047019] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049019] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.053367] rcu: Hierarchical SRCU implementation. [ 0.055749] smp: Bringing up secondary CPUs ... [ 0.056693] x86: Booting SMP configuration: [ 0.057045] .... node #0, CPUs: #1 #2 #3 [ 0.061039] smp: Brought up 1 node, 4 CPUs [ 0.063022] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228410] node 0 deferred pages initialised in 162ms [ 0.232210] devtmpfs: initialized [ 0.233305] x86/mm: Memory block size: 128MB [ 0.235897] gcov: version magic: 0x41383552 [ 0.237336] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.238105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.239445] pinctrl core: initialized pinctrl subsystem [ 0.240285] [ 0.241013] ************************************************************* [ 0.242019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243023] ** ** [ 0.244020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245020] ** ** [ 0.246020] ** This means that this kernel is built to expose internal ** [ 0.247020] ** IOMMU data structures, which may compromise security on ** [ 0.248021] ** your system. ** [ 0.249017] ** ** [ 0.250023] ** If you see this message and you are not debugging the ** [ 0.251021] ** kernel, report this immediately to your vendor! ** [ 0.252020] ** ** [ 0.253023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254021] ************************************************************* [ 0.256132] NET: Registered protocol family 16 [ 0.257570] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.258086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.259085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.260580] cpuidle: using governor menu [ 0.263114] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.266867] PCI: Using configuration type 1 for base access [ 0.268126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278066] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.281086] cryptd: max_cpu_qlen set to 1000 [ 0.284330] ACPI: Added _OSI(Module Device) [ 0.285016] ACPI: Added _OSI(Processor Device) [ 0.286020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.287016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.291161] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.294335] ACPI: Interpreter enabled [ 0.296097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.298023] ACPI: Using IOAPIC for interrupt routing [ 0.303177] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.307439] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.320249] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.322055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.325023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.329105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335860] acpiphp: Slot [2] registered [ 0.337175] acpiphp: Slot [5] registered [ 0.339212] acpiphp: Slot [6] registered [ 0.341185] acpiphp: Slot [7] registered [ 0.343138] acpiphp: Slot [8] registered [ 0.345201] acpiphp: Slot [9] registered [ 0.347168] acpiphp: Slot [10] registered [ 0.349323] acpiphp: Slot [3] registered [ 0.351196] acpiphp: Slot [4] registered [ 0.353186] acpiphp: Slot [11] registered [ 0.355293] acpiphp: Slot [12] registered [ 0.356373] acpiphp: Slot [13] registered [ 0.359135] acpiphp: Slot [14] registered [ 0.360109] acpiphp: Slot [15] registered [ 0.362164] acpiphp: Slot [16] registered [ 0.364121] acpiphp: Slot [17] registered [ 0.366129] acpiphp: Slot [18] registered [ 0.367102] acpiphp: Slot [19] registered [ 0.369089] acpiphp: Slot [20] registered [ 0.371123] acpiphp: Slot [21] registered [ 0.372200] acpiphp: Slot [22] registered [ 0.376241] acpiphp: Slot [23] registered [ 0.377113] acpiphp: Slot [24] registered [ 0.381255] acpiphp: Slot [25] registered [ 0.383127] acpiphp: Slot [26] registered [ 0.386279] acpiphp: Slot [27] registered [ 0.388284] acpiphp: Slot [28] registered [ 0.390148] acpiphp: Slot [29] registered [ 0.392126] acpiphp: Slot [30] registered [ 0.394169] acpiphp: Slot [31] registered [ 0.396085] PCI host bridge to bus 0000:00 [ 0.398031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.404030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.407036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.410035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.414046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.416278] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.420393] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.424473] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.434986] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.440041] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.452423] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.455905] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.459145] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.461980] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.468020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.483021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.488022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.495956] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.510018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.530020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.541085] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.582022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.598368] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.606020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.635021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.653699] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.661023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.669020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.686020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.713000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.722021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.732028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.764021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.776493] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.784019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.791019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.819019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831333] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.834444] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.837652] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.840435] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.842226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.846223] iommu: Default domain type: Passthrough [ 0.849316] SCSI subsystem initialized [ 0.850187] ACPI: bus type USB registered [ 0.852153] usbcore: registered new interface driver usbfs [ 0.854118] usbcore: registered new interface driver hub [ 0.856250] usbcore: registered new device driver usb [ 0.858232] pps_core: LinuxPPS API ver. 1 registered [ 0.860015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.863080] PTP clock support registered [ 0.866100] EDAC MC: Ver: 3.0.0 [ 0.868175] PCI: Using ACPI for IRQ routing [ 0.870047] NetLabel: Initializing [ 0.871011] NetLabel: domain hash size = 128 [ 0.873014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.876193] NetLabel: unlabeled traffic allowed by default [ 0.879048] vgaarb: loaded [ 0.880382] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.883019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.892133] clocksource: Switched to clocksource kvm-clock [ 1.038142] VFS: Disk quotas dquot_6.6.0 [ 1.042276] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.048159] *** VALIDATE ramfs *** [ 1.049741] *** VALIDATE hugetlbfs *** [ 1.052818] pnp: PnP ACPI init [ 1.056335] pnp: PnP ACPI: found 6 devices [ 1.076060] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.080488] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.083441] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.086279] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.088522] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.091476] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.095144] NET: Registered protocol family 2 [ 1.098195] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.106611] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.111494] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.118125] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.122327] TCP: Hash tables configured (established 65536 bind 65536) [ 1.125926] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.129632] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.132980] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.137224] NET: Registered protocol family 1 [ 1.140254] RPC: Registered named UNIX socket transport module. [ 1.143784] RPC: Registered udp transport module. [ 1.145977] RPC: Registered tcp transport module. [ 1.147922] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.150482] NET: Registered protocol family 44 [ 1.152338] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.156452] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.158699] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.161161] PCI: CLS 0 bytes, default 64 [ 1.163139] Unpacking initramfs... [ 2.674415] debug: unmapping init [mem 0xffff89ed3cc54000-0xffff89ed3ffbffff] [ 2.678856] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.681625] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.684985] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.166847] Initialise system trusted keyrings [ 3.168456] Key type blacklist registered [ 3.170156] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.178914] zbud: loaded [ 3.181540] *** VALIDATE nfs *** [ 3.182723] *** VALIDATE nfs4 *** [ 3.184240] pstore: using deflate compression [ 3.187701] Platform Keyring initialized [ 3.306317] NET: Registered protocol family 38 [ 3.307835] Key type asymmetric registered [ 3.309180] Asymmetric key parser 'x509' registered [ 3.311105] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.314113] io scheduler mq-deadline registered [ 3.316142] io scheduler kyber registered [ 3.318301] io scheduler bfq registered [ 3.321117] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.325200] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.328492] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.332089] ACPI: Power Button [PWRF] [ 3.338673] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.347410] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.363714] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.372149] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.388882] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.417940] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.446210] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.452452] Non-volatile memory driver v1.3 [ 3.454476] Linux agpgart interface v0.103 [ 3.488920] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.492282] vda: detected capacity change from 0 to 34836480 [ 3.508474] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.512021] vdb: detected capacity change from 0 to 1073741824 [ 3.527244] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.529898] vdc: detected capacity change from 0 to 2621440000 [ 3.544332] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.547055] vdd: detected capacity change from 0 to 2621440000 [ 3.561055] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.563972] vde: detected capacity change from 0 to 4294967296 [ 3.578581] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.581455] vdf: detected capacity change from 0 to 4294967296 [ 3.587397] libphy: Fixed MDIO Bus: probed [ 3.597625] usbcore: registered new interface driver usbserial_generic [ 3.599924] usbserial: USB Serial support registered for generic [ 3.602091] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.606341] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.608060] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.610149] mousedev: PS/2 mouse device common for all mice [ 3.613452] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.613901] rtc_cmos 00:05: RTC can wake from S4 [ 3.622246] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.624244] rtc_cmos 00:05: registered as rtc0 [ 3.626372] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.627702] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.633727] intel_pstate: CPU model not supported [ 3.637412] hid: raw HID events driver (C) Jiri Kosina [ 3.639931] usbcore: registered new interface driver usbhid [ 3.642215] usbhid: USB HID core driver [ 3.644103] drop_monitor: Initializing network drop monitor service [ 3.646615] Initializing XFRM netlink socket [ 3.648910] NET: Registered protocol family 10 [ 3.652246] Segment Routing with IPv6 [ 3.653726] NET: Registered protocol family 17 [ 3.655873] mpls_gso: MPLS GSO support [ 3.662213] RAS: Correctable Errors collector initialized. [ 3.665326] AVX version of gcm_enc/dec engaged. [ 3.667257] AES CTR mode by8 optimization enabled [ 3.749990] sched_clock: Marking stable (3749942871, 0)->(4821781124, -1071838253) [ 3.756704] registered taskstats version 1 [ 3.759888] Loading compiled-in X.509 certificates [ 3.762683] zswap: loaded using pool lzo/zbud [ 3.800217] Key type big_key registered [ 3.813122] Key type encrypted registered [ 3.817184] ima: No TPM chip found, activating TPM-bypass! [ 3.819279] ima: Allocated hash algorithm: sha1 [ 3.821333] ima: No architecture policies found [ 3.823279] evm: Initialising EVM extended attributes: [ 3.825424] evm: security.selinux [ 3.826504] evm: security.ima [ 3.827660] evm: security.capability [ 3.829157] evm: HMAC attrs: 0x1 [ 3.831702] rtc_cmos 00:05: setting system clock to 2026-04-14 14:49:37 UTC (1776178177) [ 3.839187] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.842439] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 3.855158] Write protecting the kernel read-only data: 28672k [ 3.859508] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.862494] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 3.910525] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.919706] systemd[1]: Detected virtualization kvm. [ 3.921692] systemd[1]: Detected architecture x86-64. [ 3.923474] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.951040] systemd[1]: No hostname configured. [ 3.952784] systemd[1]: Set hostname to . [ 3.954862] random: systemd: uninitialized urandom read (16 bytes read) [ 3.957358] systemd[1]: Initializing machine ID from random generator. [ 4.145246] random: systemd: uninitialized urandom read (16 bytes read) [ 4.149112] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.154655] random: systemd: uninitialized urandom read (16 bytes read) [ 4.157375] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.166572] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.027260] device-mapper: uevent: version 1.0.3 [ 5.029974] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.022412] virtio_net virtio0 ens2: renamed from eth0 [ 6.057162] random: fast init done [ 6.163207] scsi host0: ata_piix [ 6.199171] scsi host1: ata_piix [ 6.200982] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.203569] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.161655] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.334072] random: crng init done [ 10.335443] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.159868] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.625503] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.982577] SELinux: Disabled at runtime. [ 14.049836] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.059296] systemd[1]: Detected virtualization kvm. [ 14.061323] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.679392] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.683868] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.696529] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.701385] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.705774] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.716403] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.723975] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ 14.817347] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 15.382276] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.718207] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.727116] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.885126] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.903427] EDAC sbridge: Ver: 1.1.2 [ 18.230068] Key type dns_resolver registered [ 18.558364] NFS: Registering the id_resolver key type [ 18.560957] Key type id_resolver registered [ 18.562730] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 35.980503] hrtimer: interrupt took 6980906 ns [ 63.020265] libcfs: loading out-of-tree module taints kernel. [ 63.129567] alg: No test for adler32 (adler32-zlib) [ 63.885856] Key type ._llcrypt registered [ 63.898399] Key type .llcrypt registered [ 64.162191] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 87.185769] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 88.862323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.693787] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 90.443922] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 90.452728] LNet: Accept secure, port 988 [ 92.303168] Key type lgssc registered [ 94.113509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.835878] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.293542] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 128.898183] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.158325] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.406715] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.477604] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 185.461173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.545757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.025514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 187.093992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 187.248571] Lustre: lustre-MDT0000: new disk, initializing [ 187.436519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.475152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 193.268674] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 209.145222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.231788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.364162] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 209.421822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 209.432129] Lustre: Skipped 1 previous similar message [ 209.558692] Lustre: lustre-MDT0001: new disk, initializing [ 209.637655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.672362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.693290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.680863] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 226.669316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 226.786722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.792937] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 227.057233] Lustre: lustre-OST0000: new disk, initializing [ 227.063137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 227.127958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.028992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.034495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.659286] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 245.434704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 245.533545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.544741] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 245.703406] Lustre: lustre-OST0001: new disk, initializing [ 245.709405] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 245.802068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.733758] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 254.497086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 254.504448] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.239957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.906203] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 283.512806] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.341229] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 294.077554] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 297.071747] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 299.792229] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 301.339610] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 10:54:33 EDT 2026 [ 312.037073] Lustre: DEBUG MARKER: excepting tests: 59 [ 322.682619] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing check_config_client /mnt/lustre [ 345.891084] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 349.140849] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 353.068412] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 356.445716] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 10:55:28 (1776178528) [ 364.682387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 366.968365] Lustre: Failing over lustre-MDT0000 [ 367.079657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.092662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.096445] Lustre: Skipped 3 previous similar messages [ 367.111118] Lustre: Skipped 2 previous similar messages [ 367.335591] Lustre: server umount lustre-MDT0000 complete [ 369.586026] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.192260] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.227255] LustreError: Skipped 3 previous similar messages [ 374.704526] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.744038] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.359295] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178545/real 1776178545] req@00000000494697f6 x1862457899409408/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178552 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 379.390408] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.852589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.901446] LustreError: Skipped 8 previous similar messages [ 390.632428] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 390.647888] LustreError: Skipped 14 previous similar messages [ 395.263088] LDISKFS-fs (dm-0): recovery complete [ 395.274194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.772271] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb010f00 to 0x65248e50bb0115a5 [ 396.804371] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 397.291124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.252343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 402.432751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 402.622806] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 405.194444] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 421.521755] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 424.407252] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 438.252422] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 10:56:49 (1776178609) [ 443.016618] Lustre: Failing over lustre-OST0000 [ 443.105949] Lustre: server umount lustre-OST0000 complete [ 443.361941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.425412] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.466764] LustreError: Skipped 11 previous similar messages [ 463.022560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.197356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.536836] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 467.192277] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 471.028564] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 471.032067] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.131@tcp (at 0@lo) [ 471.047161] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 471.052393] Lustre: Skipped 3 previous similar messages [ 477.607821] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 479.022224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 487.682053] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 10:57:39 (1776178659) [ 494.891559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 496.931434] Lustre: Failing over lustre-MDT0000 [ 497.152623] Lustre: server umount lustre-MDT0000 complete [ 500.191683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.196153] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.209608] Lustre: Skipped 4 previous similar messages [ 500.237214] LustreError: Skipped 9 previous similar messages [ 506.335166] Lustre: 3369:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178673/real 1776178673] req@00000000e9343955 x1862457899446016/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178680 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 506.368920] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.404455] LDISKFS-fs (dm-0): recovery complete [ 520.443187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.784713] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0115a5 to 0x65248e50bb011fdf [ 523.832698] Lustre: MGC192.168.204.131@tcp: Connection restored to 192.168.204.131@tcp (at 0@lo) [ 523.862122] Lustre: Skipped 1 previous similar message [ 524.297794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.378858] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 529.390297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 529.615152] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 532.041362] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 537.494417] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 542.623068] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 547.728532] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 552.848805] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:42 [ 563.089919] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:31 [ 563.100732] Lustre: Skipped 1 previous similar message [ 583.576321] Lustre: lustre-MDT0000: Denying connection for new client eb45d6fd-2a95-40bf-bc0b-ccd9cb76fa70 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:11 [ 583.586352] Lustre: Skipped 3 previous similar messages [ 595.001069] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 595.007131] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 595.033423] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 595.034523] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 595.062272] Lustre: Skipped 2 previous similar messages [ 595.095628] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 595.095813] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:97 [ 611.478469] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 10:59:42 (1776178782) [ 620.830306] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 622.853517] Lustre: Failing over lustre-MDT0000 [ 623.045873] Lustre: server umount lustre-MDT0000 complete [ 626.155659] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 626.165838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.181231] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.204417] LustreError: Skipped 28 previous similar messages [ 633.824654] Lustre: 3368:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178800/real 1776178800] req@0000000093230e70 x1862457899477440/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178807 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 633.862216] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.720829] LDISKFS-fs (dm-0): recovery complete [ 646.726217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.245603] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb011fdf to 0x65248e50bb0123e4 [ 651.254302] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 651.466966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.501992] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 655.816808] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 656.871739] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 657.838365] Lustre: lustre-MDT0000: Denying connection for new client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 657.871310] Lustre: Skipped 2 previous similar messages [ 722.000291] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 722.007805] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 722.035746] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 722.036093] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 722.056253] Lustre: Skipped 3 previous similar messages [ 722.107997] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:129 [ 722.109586] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:97 [ 735.004526] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:01:46 (1776178906) [ 743.587570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 745.330990] Lustre: Failing over lustre-MDT0000 [ 745.452578] Lustre: server umount lustre-MDT0000 complete [ 748.000093] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 748.013081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.035658] Lustre: Skipped 3 previous similar messages [ 755.090804] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.110784] LustreError: Skipped 41 previous similar messages [ 756.191188] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178922/real 1776178922] req@00000000cbc5fc7e x1862457899506048/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178929 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 756.225568] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.379245] LDISKFS-fs (dm-0): recovery complete [ 769.388304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.612886] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0123e4 to 0x65248e50bb012860 [ 774.059534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.102979] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 774.550057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 779.320511] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 779.405462] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:161 [ 779.416759] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:129 [ 779.741765] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 791.128618] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 793.233574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 801.974757] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:02:53 (1776178973) [ 809.981486] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 812.133621] Lustre: Failing over lustre-MDT0000 [ 812.452601] Lustre: server umount lustre-MDT0000 complete [ 815.072062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.090809] Lustre: Skipped 5 previous similar messages [ 815.110415] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 822.239346] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178988/real 1776178988] req@00000000b52304e2 x1862457899525248/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178995 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 822.275104] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 835.978673] LDISKFS-fs (dm-0): recovery complete [ 835.980802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.626948] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb012860 to 0x65248e50bb012d5a [ 838.654785] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 838.669218] Lustre: Skipped 5 previous similar messages [ 839.085202] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 840.732527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 844.405386] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 844.411720] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 844.459501] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:193 [ 844.463090] Lustre: lustre-OST0001: deleting orphan objects from 0x0:131 to 0x0:161 [ 855.003035] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 856.475222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 864.395437] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:03:56 (1776179036) [ 871.706345] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 873.846732] Lustre: Failing over lustre-MDT0000 [ 874.027687] Lustre: server umount lustre-MDT0000 complete [ 874.979982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.997482] Lustre: Skipped 2 previous similar messages [ 882.145134] Lustre: 3368:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179048/real 1776179048] req@00000000c2c9303f x1862457899542464/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179055 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 882.164292] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.514412] LDISKFS-fs (dm-0): recovery complete [ 896.517799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.537388] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb012d5a to 0x65248e50bb01328c [ 898.915190] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 900.371642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 903.797666] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 904.346483] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 904.387952] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:225 [ 904.402152] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:193 [ 912.218844] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 913.452409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 920.993958] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:04:53 (1776179093) [ 926.628207] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 928.490194] Lustre: Failing over lustre-MDT0000 [ 928.617591] Lustre: server umount lustre-MDT0000 complete [ 929.759879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.778098] Lustre: Skipped 5 previous similar messages [ 937.951229] Lustre: 3369:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179103/real 1776179103] req@00000000c6fbed38 x1862457899558784/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179110 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 937.982446] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 951.319091] LDISKFS-fs (dm-0): recovery complete [ 951.322403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.338312] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb01328c to 0x65248e50bb0137b0 [ 954.643873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.649248] Lustre: Skipped 2 previous similar messages [ 954.726216] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 954.772568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 958.555178] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 960.072185] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 960.100941] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 960.101719] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:225 [ 966.123412] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 967.539206] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 973.931440] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:05:46 (1776179146) [ 980.281703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 992.735372] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179159/real 1776179159] req@00000000e5381e7c x1862457899575360/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179166 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 992.757902] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.322625] LDISKFS-fs (dm-0): recovery complete [ 1004.330519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.147805] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0137b0 to 0x65248e50bb013cf7 [ 1010.170524] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 1010.179167] Lustre: Skipped 14 previous similar messages [ 1010.408168] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1014.243960] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1015.873420] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:257 [ 1015.877446] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:289 [ 1024.458777] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1026.302320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1034.775715] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:06:46 (1776179206) [ 1035.859236] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1035.862764] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1035.865583] LustreError: 6276:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000770071b5 x1862457876357440/t38654705666(0) o35->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:190/0 lens 392/456 e 0 to 0 dl 1776179215 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 1043.408554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1043.423088] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 1043.481753] Lustre: 6276:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000e9e812e7 x1862457876357440/t38654705666(0) o35->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:198/0 lens 392/456 e 0 to 0 dl 1776179223 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 1044.886938] Lustre: Failing over lustre-MDT0000 [ 1044.890417] Lustre: Skipped 1 previous similar message [ 1046.497231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.499105] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.525874] Lustre: Skipped 6 previous similar messages [ 1046.553619] LustreError: Skipped 178 previous similar messages [ 1047.089241] Lustre: server umount lustre-MDT0000 complete [ 1047.095956] Lustre: Skipped 1 previous similar message [ 1069.691646] LDISKFS-fs (dm-0): recovery complete [ 1069.696490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.394155] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1072.021771] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1072.031721] Lustre: Skipped 1 previous similar message [ 1075.622369] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1076.765261] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1076.777696] Lustre: Skipped 1 previous similar message [ 1076.823446] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:289 [ 1076.825724] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:321 [ 1086.026793] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1087.761665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1095.851563] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:07:47 (1776179267) [ 1102.217147] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1107.430436] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1114.463608] Lustre: 3368:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179281/real 1776179281] req@0000000034ad8da5 x1862457899611776/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179288 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1114.489698] Lustre: 3368:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1114.503627] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1114.517867] LustreError: Skipped 1 previous similar message [ 1125.580581] LDISKFS-fs (dm-0): recovery complete [ 1125.585208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.006108] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0141c7 to 0x65248e50bb014674 [ 1132.014170] Lustre: Skipped 1 previous similar message [ 1132.322035] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1135.866687] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1137.692526] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:353 [ 1137.692624] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:321 [ 1142.910873] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.129204] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1150.809358] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:08:42 (1776179322) [ 1158.173803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1159.255280] Lustre: *** cfs_fail_loc=114, val=0*** [ 1163.236203] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1188.248870] LDISKFS-fs (dm-0): recovery complete [ 1188.251563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.425779] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1204.507321] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1204.519677] Lustre: Skipped 1 previous similar message [ 1207.542288] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1208.861399] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1208.883191] Lustre: Skipped 1 previous similar message [ 1208.939134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:385 [ 1208.941596] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:353 [ 1216.789551] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1218.006819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1224.819789] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:09:56 (1776179396) [ 1231.790464] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1232.638348] Lustre: *** cfs_fail_loc=128, val=0*** [ 1235.164093] Lustre: Failing over lustre-MDT0000 [ 1235.169388] Lustre: Skipped 2 previous similar messages [ 1235.317719] Lustre: server umount lustre-MDT0000 complete [ 1235.327461] Lustre: Skipped 2 previous similar messages [ 1239.521439] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1239.526218] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.554315] Lustre: Skipped 12 previous similar messages [ 1247.199310] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179413/real 1776179413] req@0000000027bd5e59 x1862457899649600/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179420 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1247.228969] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1247.246608] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.268985] LustreError: Skipped 1 previous similar message [ 1257.323038] LDISKFS-fs (dm-0): recovery complete [ 1257.328759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.609379] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb014b60 to 0x65248e50bb01501b [ 1264.614139] Lustre: Skipped 1 previous similar message [ 1264.819476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.823763] Lustre: Skipped 4 previous similar messages [ 1268.616584] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1270.253816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1270.265256] Lustre: Skipped 20 previous similar messages [ 1270.330620] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:385 [ 1270.332057] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:417 [ 1276.343220] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1277.655619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1284.720721] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:10:56 (1776179456) [ 1292.177847] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1295.843331] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.472291] LDISKFS-fs (dm-0): recovery complete [ 1317.480215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.041313] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1325.785301] Lustre: lustre-OST0000: deleting orphan objects from 0x0:423 to 0x0:449 [ 1325.785603] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 1331.188184] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1332.661886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1339.921696] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:11:51 (1776179511) [ 1345.855477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1369.830311] LDISKFS-fs (dm-0): recovery complete [ 1369.834570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.964095] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1374.969238] Lustre: Skipped 2 previous similar messages [ 1378.907728] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1380.538116] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 1380.538469] Lustre: lustre-OST0001: deleting orphan objects from 0x0:423 to 0x0:449 [ 1387.657974] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1389.288095] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1397.322118] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:12:48 (1776179568) [ 1405.006467] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1411.554417] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1411.572082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1411.579765] Lustre: Skipped 1 previous similar message [ 1435.009565] LDISKFS-fs (dm-0): recovery complete [ 1435.019583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.989562] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1448.561366] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:609 [ 1448.563067] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:577 [ 1452.984255] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1454.199430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1475.331852] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:14:07 (1776179647) [ 1481.908086] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1484.773118] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1504.649030] LDISKFS-fs (dm-0): recovery complete [ 1504.652572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.294570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1510.306061] Lustre: Skipped 4 previous similar messages [ 1514.487543] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1515.584654] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1515.593812] Lustre: Skipped 4 previous similar messages [ 1515.627712] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:609 [ 1515.632733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:641 [ 1524.744309] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1526.446886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1537.009120] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:15:08 (1776179708) [ 1544.251771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1546.074986] Lustre: Failing over lustre-MDT0000 [ 1546.077746] Lustre: Skipped 4 previous similar messages [ 1546.142779] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 1546.210421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.236921] Lustre: Skipped 18 previous similar messages [ 1546.281733] Lustre: server umount lustre-MDT0000 complete [ 1546.285082] Lustre: Skipped 4 previous similar messages [ 1557.472020] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179724/real 1776179724] req@000000004419c2d9 x1862457899775296/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179731 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1557.495446] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 1557.510056] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1557.528715] LustreError: Skipped 4 previous similar messages [ 1562.592467] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1562.620129] LustreError: Skipped 342 previous similar messages [ 1572.056414] LDISKFS-fs (dm-0): recovery complete [ 1572.066351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.912977] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb01eec1 to 0x65248e50bb01f3f3 [ 1574.935405] Lustre: Skipped 4 previous similar messages [ 1579.809560] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1580.591782] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:673 [ 1580.592333] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:641 [ 1587.429259] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1588.440396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1594.459933] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:16:06 (1776179766) [ 1599.457049] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1624.277762] LDISKFS-fs (dm-0): recovery complete [ 1624.283237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.677964] LustreError: 50034:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.131@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000ae6f85c2) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1633.644561] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1635.413574] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:705 [ 1635.415813] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:673 [ 1642.131397] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1643.555814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1650.923740] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:17:02 (1776179822) [ 1656.688877] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1660.895666] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1660.909966] LustreError: Skipped 2 previous similar messages [ 1679.116811] LDISKFS-fs (dm-0): recovery complete [ 1679.119775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.645379] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1684.661054] Lustre: Skipped 4 previous similar messages [ 1688.149896] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1690.210172] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:705 [ 1690.210709] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:737 [ 1696.153575] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1697.546172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1704.093926] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:17:56 (1776179876) [ 1708.435781] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1731.517655] LDISKFS-fs (dm-0): recovery complete [ 1731.521061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.449356] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1740.877174] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:769 [ 1740.880822] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:737 [ 1749.777880] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.621690] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1759.568307] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:18:51 (1776179931) [ 1766.244757] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1790.000264] LDISKFS-fs (dm-0): recovery complete [ 1790.010363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.093723] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 1796.102138] Lustre: Skipped 43 previous similar messages [ 1796.312917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.318491] Lustre: Skipped 8 previous similar messages [ 1800.688259] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1801.852703] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:801 [ 1801.856517] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:769 [ 1812.214499] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.416112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1822.907717] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:19:54 (1776179994) [ 1830.192898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1854.000200] LDISKFS-fs (dm-0): recovery complete [ 1854.010851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.831893] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1862.780318] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:833 [ 1870.955184] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1872.733846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1881.114883] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:20:52 (1776180052) [ 1888.881948] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1912.719797] LDISKFS-fs (dm-0): recovery complete [ 1912.724204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.154988] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1923.187299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:865 [ 1923.195984] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1932.361411] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1934.310633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1944.202743] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:21:55 (1776180115) [ 1952.771803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1958.890729] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1958.897190] LustreError: Skipped 1 previous similar message [ 1978.671111] LDISKFS-fs (dm-0): recovery complete [ 1978.674272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.029385] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1988.713153] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:833 [ 1988.715064] Lustre: lustre-OST0000: deleting orphan objects from 0x0:867 to 0x0:897 [ 1998.342545] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2000.040697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2008.248516] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:22:59 (1776180179) [ 2015.470934] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2038.275536] LDISKFS-fs (dm-0): recovery complete [ 2038.278714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.806959] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2043.828148] Lustre: Skipped 8 previous similar messages [ 2047.467597] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2048.562643] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2048.567870] Lustre: Skipped 8 previous similar messages [ 2048.633400] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:865 [ 2048.639798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:867 to 0x0:929 [ 2060.389959] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2062.699833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2072.656723] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:24:04 (1776180244) [ 2081.083510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2082.907760] Lustre: Failing over lustre-MDT0000 [ 2082.910897] Lustre: Skipped 8 previous similar messages [ 2083.071842] Lustre: server umount lustre-MDT0000 complete [ 2083.074760] Lustre: Skipped 8 previous similar messages [ 2084.320569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.330861] Lustre: Skipped 35 previous similar messages [ 2090.463326] Lustre: 3369:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180257/real 1776180257] req@000000002fc222ca x1862457899936448/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180264 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2090.496032] Lustre: 3369:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2090.501184] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.509106] LustreError: Skipped 8 previous similar messages [ 2105.620253] LDISKFS-fs (dm-0): recovery complete [ 2105.630300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.877452] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb021e4e to 0x65248e50bb02239c [ 2107.901228] Lustre: Skipped 8 previous similar messages [ 2112.255258] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2113.748414] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:961 [ 2113.750630] Lustre: lustre-OST0001: deleting orphan objects from 0x0:867 to 0x0:897 [ 2122.832461] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2124.651489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2133.699200] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:25:05 (1776180305) [ 2140.289506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2162.238176] LDISKFS-fs (dm-0): recovery complete [ 2162.242094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.266230] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.279658] LustreError: Skipped 395 previous similar messages [ 2173.494995] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2174.061139] Lustre: lustre-OST0001: deleting orphan objects from 0x0:867 to 0x0:929 [ 2174.061194] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:993 [ 2185.240935] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2187.535553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2197.376457] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:26:08 (1776180368) [ 2205.321113] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2226.756430] LDISKFS-fs (dm-0): recovery complete [ 2226.759243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.038926] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2227.045434] Lustre: Skipped 8 previous similar messages [ 2230.480918] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2232.341513] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:1025 [ 2232.342537] Lustre: lustre-OST0001: deleting orphan objects from 0x0:931 to 0x0:961 [ 2238.344941] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2239.722947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2246.340421] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:26:58 (1776180418) [ 2252.855858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2275.136278] LDISKFS-fs (dm-0): recovery complete [ 2275.139316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.744794] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2286.720487] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2286.721370] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 2292.938786] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2294.293349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2302.016744] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:27:53 (1776180473) [ 2309.950673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2312.672560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.694463] Lustre: Skipped 4 previous similar messages [ 2314.161039] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 2336.544558] LDISKFS-fs (dm-0): recovery complete [ 2336.562460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.927928] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2347.134051] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 2347.134915] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2354.986632] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2356.302459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2363.968405] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:28:55 (1776180535) [ 2371.596259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2394.467230] LDISKFS-fs (dm-0): recovery complete [ 2394.471596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.297122] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2400.231868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2400.244938] Lustre: Skipped 50 previous similar messages [ 2400.328255] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2400.332453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1121 [ 2406.602677] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2407.956479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2414.622553] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:29:46 (1776180586) [ 2417.864708] Lustre: 77585:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 at adminstrative request [ 2443.768138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.598732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.606889] Lustre: Skipped 10 previous similar messages [ 2453.229370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2455.073364] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1089 [ 2455.077250] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2460.969165] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2462.298366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2467.961965] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2475.910307] Lustre: DEBUG MARKER: before 2844, after 2844 [ 2481.358192] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:30:53 (1776180653) [ 2482.456958] Lustre: 79635:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 at adminstrative request [ 2494.785156] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:31:05 (1776180665) [ 2502.867782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2506.209791] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2506.220368] LustreError: Skipped 5 previous similar messages [ 2526.828543] LDISKFS-fs (dm-0): recovery complete [ 2526.832549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.129174] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2535.528620] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2535.544352] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1156 to 0x0:1185 [ 2543.787233] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2545.537789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2554.645729] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:32:06 (1776180726) [ 2562.531333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2586.978462] LDISKFS-fs (dm-0): recovery complete [ 2586.981571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.739802] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2595.884276] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2595.887192] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2600.837878] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2602.312052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2609.515580] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:33:01 (1776180781) [ 2616.564745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2638.447851] LDISKFS-fs (dm-0): recovery complete [ 2638.451042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.977661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2646.985536] Lustre: Skipped 9 previous similar messages [ 2649.398110] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2651.664117] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2651.669516] Lustre: Skipped 9 previous similar messages [ 2651.689064] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2651.690737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2656.983774] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2658.594685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2666.272481] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:33:58 (1776180838) [ 2672.887615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2695.576153] LDISKFS-fs (dm-0): recovery complete [ 2695.579312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.647533] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2706.463790] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2706.463789] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2712.030547] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2713.191036] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2718.953918] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:34:51 (1776180891) [ 2723.840931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2725.101085] Lustre: Failing over lustre-MDT0000 [ 2725.103488] Lustre: Skipped 10 previous similar messages [ 2725.224067] Lustre: server umount lustre-MDT0000 complete [ 2725.227386] Lustre: Skipped 10 previous similar messages [ 2726.880669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.889228] Lustre: Skipped 43 previous similar messages [ 2734.559203] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180900/real 1776180900] req@00000000d9e56bb2 x1862457900130432/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180907 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2734.574374] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2734.580490] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.588759] LustreError: Skipped 10 previous similar messages [ 2744.446758] LDISKFS-fs (dm-0): recovery complete [ 2744.450722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.970613] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb025a30 to 0x65248e50bb025f93 [ 2751.980077] Lustre: Skipped 10 previous similar messages [ 2755.357389] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2757.658211] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2757.659422] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1313 [ 2763.419574] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.684602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2771.593666] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:35:43 (1776180943) [ 2778.130280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2783.200997] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.228557] LustreError: Skipped 352 previous similar messages [ 2804.134914] LDISKFS-fs (dm-0): recovery complete [ 2804.139125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.952755] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2812.599825] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2812.600676] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2821.263876] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2822.827179] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2831.386480] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:36:43 (1776181003) [ 2838.128350] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2861.885334] LDISKFS-fs (dm-0): recovery complete [ 2861.897847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.871794] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2866.874962] Lustre: Skipped 10 previous similar messages [ 2871.218674] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2872.565627] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2872.568723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2880.323792] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2882.243591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2892.014161] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:37:43 (1776181063) [ 2900.832982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2926.091738] LDISKFS-fs (dm-0): recovery complete [ 2926.094453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.379616] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2932.331428] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2932.332465] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2939.536123] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2941.208620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2949.211955] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:38:40 (1776181120) [ 2955.742843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2979.043614] LDISKFS-fs (dm-0): recovery complete [ 2979.046909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.588477] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2993.767108] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2993.774032] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 3002.823766] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3004.934260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3014.764607] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:39:46 (1776181186) [ 3022.683418] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3024.799236] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 3047.756579] LDISKFS-fs (dm-0): recovery complete [ 3047.760571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.059638] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 3054.069472] Lustre: Skipped 53 previous similar messages [ 3054.251111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.255086] Lustre: Skipped 9 previous similar messages [ 3058.096827] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3059.747914] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 3059.748244] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 3065.912792] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3067.324746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3074.786446] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:40:46 (1776181246) [ 3080.959181] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3103.995500] LDISKFS-fs (dm-0): recovery complete [ 3103.998517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.257161] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3114.537964] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 3114.538886] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 3120.762597] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3122.103486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3129.207752] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:41:41 (1776181301) [ 3130.162351] Lustre: 102317:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 at adminstrative request [ 3137.567851] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:41:49 (1776181309) [ 3142.801297] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3145.185508] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3145.192848] LustreError: Skipped 4 previous similar messages [ 3154.677842] LDISKFS-fs (dm-0): recovery complete [ 3154.680846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.728930] LustreError: 103598:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3158.736658] LustreError: 103598:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3158.743809] Lustre: 103632:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3158.753069] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3158.762191] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3158.774855] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3158.823493] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1537 [ 3158.829995] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1473 [ 3163.172601] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3164.146810] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3183.272791] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 11:42:34 (1776181354) [ 3190.105827] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3203.343238] LDISKFS-fs (dm-0): recovery complete [ 3203.345788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.484169] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3208.516326] LustreError: 105517:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3208.525057] LustreError: 105517:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3208.533673] Lustre: 105551:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3208.543040] Lustre: 105551:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3208.548568] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3208.563951] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3208.591812] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3208.656386] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1479 to 0x0:1505 [ 3208.667310] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1543 to 0x0:1569 [ 3212.473540] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3213.794996] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3225.293282] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3225.296878] Lustre: Skipped 1 previous similar message [ 3230.017329] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:43:22 (1776181402) [ 3237.060408] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3250.967302] LDISKFS-fs (dm-0): recovery complete [ 3250.983502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.072850] LustreError: 107436:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3253.086211] LustreError: 107436:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3253.094771] Lustre: 107469:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3253.103894] Lustre: 107469:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3253.113130] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3253.125470] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3253.174337] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3253.328698] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1576 to 0x0:1601 [ 3253.330230] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1511 to 0x0:1537 [ 3258.356989] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3259.018504] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3278.834941] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:44:10 (1776181450) [ 3279.510890] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3279.513126] LustreError: 6275:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000072f22547 x1862457876891136/t201863462916(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:203/0 lens 504/456 e 0 to 0 dl 1776181493 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 3290.882703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.034577] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (not set up) [ 3291.284820] LustreError: 108812:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3291.297250] LustreError: 108812:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3291.306396] Lustre: 108845:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3291.326813] Lustre: 108845:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3291.339751] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3291.368124] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3291.402777] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3291.453408] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1576 to 0x0:1633 [ 3291.464311] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1569 [ 3295.602635] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3296.226575] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3315.114200] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 11:44:46 (1776181486) [ 3322.779985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3326.842315] Lustre: server umount lustre-MDT0000 complete [ 3326.850728] Lustre: Skipped 10 previous similar messages [ 3326.946277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.967755] Lustre: Skipped 43 previous similar messages [ 3347.506440] LDISKFS-fs (dm-0): recovery complete [ 3347.509691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.443527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3351.451067] Lustre: Skipped 8 previous similar messages [ 3354.413857] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3356.721820] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3356.732301] Lustre: Skipped 8 previous similar messages [ 3356.784884] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1665 [ 3356.786305] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1601 [ 3362.378281] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:45:34 (1776181534) [ 3368.015568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3370.301817] Lustre: Failing over lustre-MDT0000 [ 3370.303619] Lustre: Skipped 11 previous similar messages [ 3379.679978] Lustre: 3369:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181545/real 1776181545] req@00000000c0f1f61b x1862457900335552/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181552 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3379.711453] Lustre: 3369:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 3379.717739] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3379.727166] LustreError: Skipped 11 previous similar messages [ 3381.976179] LDISKFS-fs (dm-0): recovery complete [ 3381.980835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.221233] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.253753] LustreError: Skipped 364 previous similar messages [ 3385.836034] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb02a283 to 0x65248e50bb02a7df [ 3385.861514] Lustre: Skipped 11 previous similar messages [ 3386.129777] LustreError: 112577:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3386.136201] LustreError: 112577:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3386.143204] Lustre: 112611:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3386.152461] Lustre: 112611:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3386.157824] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3386.165791] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3386.190549] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3386.246593] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1633 [ 3386.268207] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1697 [ 3389.847697] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3391.457409] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3407.532564] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:46:19 (1776181579) [ 3437.686662] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3466.026089] LDISKFS-fs (dm-0): recovery complete [ 3466.029435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.503961] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3467.508539] Lustre: Skipped 20 previous similar messages [ 3470.571736] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3472.973972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113 [ 3472.977856] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049 [ 3477.888327] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3479.230466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3494.176921] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:47:46 (1776181666) [ 3513.430757] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3542.989045] LDISKFS-fs (dm-0): recovery complete [ 3542.992134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.322820] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3555.419432] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465 [ 3555.419875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 3559.757823] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3560.984542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3576.380957] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:49:08 (1776181748) [ 3577.419509] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3578.703675] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:49:10 (1776181750) [ 3580.471282] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3581.334721] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3581.341493] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3581.354876] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2497 [ 3585.920251] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:49:18 (1776181758) [ 3603.912855] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3634.720146] LDISKFS-fs (dm-2): recovery complete [ 3634.734186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3636.953188] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3636.958619] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977 [ 3638.662808] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3697.594790] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:51:09 (1776181869) [ 3704.657359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3728.140551] LDISKFS-fs (dm-0): recovery complete [ 3728.145638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.531988] Lustre: MGC192.168.204.131@tcp: Connection restored to 192.168.204.131@tcp (at 0@lo) [ 3735.555381] Lustre: Skipped 52 previous similar messages [ 3735.794140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.797567] Lustre: Skipped 10 previous similar messages [ 3739.823573] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3741.222679] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3741.226137] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009 [ 3748.096331] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3748.319576] LustreError: 122398:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3748.324214] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3749.351613] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2913 [ 3749.801912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3770.302231] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:52:21 (1776181941) [ 3775.680205] LustreError: 8680:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3781.091445] LustreError: 8680:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3781.098110] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3781.149637] LustreError: 44571:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3782.701467] LustreError: 64694:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3787.743136] LustreError: 64694:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3787.749213] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3789.172138] LustreError: 6275:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3794.399614] LustreError: 6275:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3794.411942] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3795.991743] LustreError: 6275:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3801.056505] LustreError: 6275:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3803.027080] LustreError: 6273:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3808.224720] LustreError: 6273:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3808.230586] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3808.238143] Lustre: Skipped 1 previous similar message [ 3816.317692] LustreError: 6273:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3816.322829] LustreError: 6273:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3821.535254] LustreError: 6273:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3821.540563] LustreError: 6273:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3828.115925] LustreError: 64694:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3828.122163] LustreError: 64694:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3828.131431] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3828.132954] Lustre: lustre-MDT0000: Export 00000000bc71d736 already connecting from 192.168.204.31@tcp [ 3828.141339] Lustre: Skipped 2 previous similar messages [ 3834.864842] LustreError: 64694:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3836.488146] LustreError: 8681:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3836.497535] LustreError: 8681:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3841.503120] LustreError: 8681:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3841.510527] LustreError: 8681:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3849.437450] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:53:41 (1776182021) [ 3851.382187] LustreError: 6275:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3857.818440] Lustre: lustre-MDT0000: Export 00000000bc71d736 already connecting from 192.168.204.31@tcp [ 3859.817580] Lustre: lustre-MDT0000: Export 00000000bc71d736 already connecting from 192.168.204.31@tcp [ 3862.901428] Lustre: lustre-MDT0000: Export 00000000bc71d736 already connecting from 192.168.204.31@tcp [ 3862.905209] Lustre: Skipped 1 previous similar message [ 3867.031313] Lustre: lustre-MDT0000: Export 00000000bc71d736 already connecting from 192.168.204.31@tcp [ 3867.034837] Lustre: Skipped 1 previous similar message [ 3875.727700] LustreError: 6275:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3875.746553] Lustre: 6275:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/4s); client may timeout req@0000000056323054 x1862457877919424/t0(0) o38->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1776182045 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3878.293584] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3878.307398] Lustre: Skipped 4 previous similar messages [ 3879.935404] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:54:11 (1776182051) [ 3885.141186] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3889.633333] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3889.643623] LustreError: Skipped 9 previous similar messages [ 3897.108307] LDISKFS-fs (dm-0): recovery complete [ 3897.110892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.338073] Lustre: *** cfs_fail_loc=712, val=0*** [ 3897.342782] LustreError: 44567:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@000000003cccb8f6 x1862457900660096/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3897.360458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3897.422500] LustreError: 127071:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3897.426297] LustreError: 127071:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3897.431053] Lustre: 127105:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3897.435821] Lustre: 127105:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3897.439913] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3897.445363] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3897.461116] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3897.503482] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2945 [ 3897.511208] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041 [ 3900.374929] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3902.441357] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3933.824073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.578938] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3947.067247] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073 [ 3947.069117] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2977 [ 3956.141326] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3957.833325] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3964.712813] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 11:55:36 (1776182136) [ 3964.805457] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 3971.100722] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 11:55:43 (1776182143) [ 3971.832653] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3971.842065] LustreError: 6283:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000001483ebef x1862457877945280/t0(0) o700->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:106/0 lens 264/248 e 0 to 0 dl 1776182151 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 3980.090313] Lustre: Failing over lustre-MDT0000 [ 3980.093384] Lustre: Skipped 6 previous similar messages [ 3980.238648] Lustre: server umount lustre-MDT0000 complete [ 3980.240975] Lustre: Skipped 7 previous similar messages [ 3982.816432] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.833758] Lustre: Skipped 31 previous similar messages [ 3987.936269] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.950201] LustreError: Skipped 196 previous similar messages [ 3989.983231] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182156/real 1776182156] req@00000000ac340f1c x1862457900686464/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182163 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 3990.011530] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 3990.018740] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3990.027817] LustreError: Skipped 5 previous similar messages [ 3996.861123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.369844] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb05d2c0 to 0x65248e50bb05d751 [ 4006.373595] Lustre: Skipped 5 previous similar messages [ 4006.800742] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4006.809901] Lustre: Skipped 5 previous similar messages [ 4009.394097] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4012.025783] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4012.035105] Lustre: Skipped 5 previous similar messages [ 4012.074781] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2979 to 0x0:3009 [ 4012.074782] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105 [ 4015.812699] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4016.985550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4024.296255] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:56:36 (1776182196) [ 4046.811419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.720142] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 4048.736017] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137 [ 4051.494247] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4060.276723] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4061.561953] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4130.202168] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 11:58:22 (1776182302) [ 4135.359535] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4158.483387] LDISKFS-fs (dm-0): recovery complete [ 4158.490430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.531517] LustreError: 133531:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.131@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000001a37e455) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4162.792708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4162.797966] Lustre: Skipped 9 previous similar messages [ 4166.053930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4168.570045] Lustre: *** cfs_fail_loc=216, val=0*** [ 4168.571813] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3030 to 0x0:3073 [ 4168.573566] LustreError: 133541:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4169.631645] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 4234.294136] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:00:06 (1776182406) [ 4235.487201] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4235.495467] Lustre: Skipped 2 previous similar messages [ 4235.500811] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201 [ 4236.138865] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233 [ 4246.006789] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:00:17 (1776182417) [ 4266.327456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.208584] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4279.304757] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4279.330203] LustreError: 135227:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000487ce84a x1862457878000896/t0(0) o101->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:442/0 lens 328/344 e 0 to 0 dl 1776182487 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4318.100349] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 4318.180657] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3084 to 0x0:3105 [ 4318.181168] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 4323.646581] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4324.739878] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4331.899410] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:01:43 (1776182503) [ 4334.059347] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4342.251583] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4364.197782] LDISKFS-fs (dm-0): recovery complete [ 4364.204393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4369.382044] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 4369.400935] Lustre: Skipped 35 previous similar messages [ 4369.556620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.564244] Lustre: Skipped 6 previous similar messages [ 4373.364621] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4375.187956] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3297 [ 4375.203859] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137 [ 4381.290411] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4382.741840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4389.373314] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:02:41 (1776182561) [ 4390.212183] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4399.118092] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4427.174269] LDISKFS-fs (dm-0): recovery complete [ 4427.183016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.324232] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4435.967510] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329 [ 4435.969347] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3169 [ 4442.984082] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.458600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4451.732982] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:03:43 (1776182623) [ 4452.703349] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4460.270276] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4483.642966] LDISKFS-fs (dm-0): recovery complete [ 4483.649068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.350562] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4496.924064] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 4496.924217] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3201 [ 4507.264763] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:04:38 (1776182678) [ 4509.844520] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4509.846066] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4509.855702] LustreError: 43098:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000050acf870 x1862457878033600/t261993005073(0) o35->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:644/0 lens 392/456 e 0 to 0 dl 1776182689 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4517.343733] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4517.358459] LustreError: Skipped 7 previous similar messages [ 4533.036306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.356150] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4547.601124] Lustre: 43098:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000cab4cad6 x1862457878033600/t261993005073(0) o35->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:682/0 lens 392/456 e 0 to 0 dl 1776182727 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4547.610115] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 4547.611405] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3233 [ 4555.458592] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4556.858486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4565.590845] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:05:37 (1776182737) [ 4566.609615] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4566.614120] LustreError: 64694:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000ce319b10 x1862457878042752/t266287972368(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:735/0 lens 504/448 e 0 to 0 dl 1776182780 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4576.058615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4588.512836] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.530283] LustreError: Skipped 277 previous similar messages [ 4590.559444] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182757/real 1776182757] req@0000000082b73d34 x1862457900856256/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182764 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 4590.608685] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4590.619186] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4590.624470] LustreError: Skipped 6 previous similar messages [ 4602.245559] LDISKFS-fs (dm-0): recovery complete [ 4602.248062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.971295] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb061309 to 0x65248e50bb061999 [ 4607.981990] Lustre: Skipped 6 previous similar messages [ 4608.420786] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4608.445766] Lustre: Skipped 7 previous similar messages [ 4613.654526] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4613.665509] Lustre: Skipped 7 previous similar messages [ 4613.714812] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 4613.720499] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3425 [ 4613.758483] Lustre: 6275:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000005c995c5f x1862457878042752/t266287972368(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:27/0 lens 504/448 e 0 to 0 dl 1776182827 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4613.954693] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4624.144835] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4626.019083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4634.870291] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:06:46 (1776182806) [ 4636.154526] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4636.161413] LustreError: 6273:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000081f9f300 x1862457878052352/t270582939664(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:49/0 lens 504/448 e 0 to 0 dl 1776182849 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4638.260856] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4645.778634] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 4645.787308] Lustre: Skipped 2 previous similar messages [ 4645.799369] Lustre: 6273:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ea8d5a23 x1862457878052352/t270582939664(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:59/0 lens 504/448 e 0 to 0 dl 1776182859 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4645.820654] Lustre: 6273:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4646.897575] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4648.836510] Lustre: Failing over lustre-MDT0000 [ 4648.839348] Lustre: Skipped 8 previous similar messages [ 4649.052136] Lustre: server umount lustre-MDT0000 complete [ 4649.058399] Lustre: Skipped 8 previous similar messages [ 4649.441156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.461312] Lustre: Skipped 35 previous similar messages [ 4673.699224] LDISKFS-fs (dm-0): recovery complete [ 4673.709106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4679.714491] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457 [ 4679.718206] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3297 [ 4680.645370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4692.721554] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:07:44 (1776182864) [ 4693.789759] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4693.792411] Lustre: Skipped 1 previous similar message [ 4693.795384] LustreError: 8680:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000f51d2674 x1862457878060672/t274877906960(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:107/0 lens 504/448 e 0 to 0 dl 1776182907 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4693.816588] LustreError: 8680:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4695.694069] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4695.701637] Lustre: Skipped 1 previous similar message [ 4702.113941] Lustre: 64694:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000009cc96b7d x1862457878060672/t274877906960(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:115/0 lens 504/448 e 0 to 0 dl 1776182915 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4705.337257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4707.265169] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 4732.102596] LDISKFS-fs (dm-0): recovery complete [ 4732.111102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.046624] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4740.765741] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489 [ 4740.780893] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3329 [ 4750.958099] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:08:42 (1776182922) [ 4752.088729] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4754.005040] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4754.013126] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4754.024696] LustreError: 43098:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000002f9a48ea x1862457878069440/t279172874256(0) o35->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:133/0 lens 392/456 e 0 to 0 dl 1776182933 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4761.573575] Lustre: 43098:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000001edd961 x1862457878069440/t279172874256(0) o35->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:141/0 lens 392/456 e 0 to 0 dl 1776182941 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4763.197514] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4789.245545] LDISKFS-fs (dm-0): recovery complete [ 4789.253783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.156975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4790.172986] Lustre: Skipped 8 previous similar messages [ 4794.936370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4795.451994] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3361 [ 4795.457684] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521 [ 4808.106501] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:09:39 (1776182979) [ 4809.277675] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4809.285970] LustreError: 6273:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000008fbdf9 x1862457878076608/t283467841549(0) o101->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:222/0 lens 664/600 e 0 to 0 dl 1776183022 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4850.636573] Lustre: 6274:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000012b9ea03 x1862457878076608/t283467841549(0) o101->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:264/0 lens 664/3424 e 0 to 0 dl 1776183064 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 4859.875864] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:10:31 (1776183031) [ 4868.527810] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4894.300965] LDISKFS-fs (dm-0): recovery complete [ 4894.307037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.165690] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4902.994733] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 4902.994855] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3553 [ 4911.218910] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4912.920649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4931.315621] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:11:42 (1776183102) [ 4939.849862] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4966.714209] LDISKFS-fs (dm-0): recovery complete [ 4966.718337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.474418] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 4970.484617] Lustre: Skipped 44 previous similar messages [ 4970.699857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.708981] Lustre: Skipped 8 previous similar messages [ 4974.285534] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4976.175608] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3395 to 0x0:3425 [ 4976.175608] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3585 [ 4981.717295] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4983.121708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4988.137506] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4998.193881] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:12:50 (1776183170) [ 5040.814442] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5042.656801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5042.663278] Lustre: Skipped 2 previous similar messages [ 5044.127062] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 5064.095822] LDISKFS-fs (dm-0): recovery complete [ 5064.098571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.552664] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5078.137646] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4705 [ 5078.138397] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865 [ 5083.438137] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5084.897236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5145.519100] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:15:17 (1776183317) [ 5152.912710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5175.362994] LDISKFS-fs (dm-0): recovery complete [ 5175.371686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.383367] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5189.740595] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4867 to 0x0:4897 [ 5189.744409] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4737 [ 5195.914563] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5197.280833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5205.691453] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5207.124703] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5212.279506] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:16:24 (1776183384) [ 5219.381526] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5260.177484] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 5260.193555] Lustre: Skipped 3 previous similar messages [ 5263.053133] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5263.055560] Lustre: Skipped 1 previous similar message [ 5263.064475] LustreError: 6274:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000002b821e55 x1862457879568128/t300647710728(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:676/0 lens 66040/440 e 0 to 0 dl 1776183476 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5305.241756] Lustre: 64694:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000019b5bc8 x1862457879568128/t300647710728(0) o36->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:718/0 lens 66040/440 e 0 to 0 dl 1776183518 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5314.439693] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5316.556343] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:18:07 (1776183487) [ 5330.058356] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5332.683773] Lustre: Failing over lustre-MDT0000 [ 5332.687019] Lustre: Skipped 6 previous similar messages [ 5332.849328] Lustre: server umount lustre-MDT0000 complete [ 5332.853875] Lustre: Skipped 6 previous similar messages [ 5332.962028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.963652] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.964646] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5332.964654] LustreError: Skipped 5 previous similar messages [ 5332.977838] Lustre: Skipped 28 previous similar messages [ 5333.022026] LustreError: Skipped 317 previous similar messages [ 5345.240986] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183511/real 1776183511] req@0000000098b7c61a x1862457901424960/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183518 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5345.266728] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 5345.270257] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5345.276746] LustreError: Skipped 7 previous similar messages [ 5356.278293] LDISKFS-fs (dm-0): recovery complete [ 5356.282706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.636232] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0a44b0 to 0x65248e50bb0a72f4 [ 5361.648646] Lustre: Skipped 7 previous similar messages [ 5363.592717] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5363.602344] Lustre: Skipped 7 previous similar messages [ 5366.521140] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5368.429673] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5368.437951] Lustre: Skipped 7 previous similar messages [ 5368.475965] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4998 to 0x0:5025 [ 5368.480179] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4839 to 0x0:4865 [ 5375.854390] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.353141] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5387.455747] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:19:18 (1776183558) [ 5409.473814] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5441.577063] LDISKFS-fs (dm-2): recovery complete [ 5441.580600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5441.725620] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5441.756777] Lustre: Skipped 5 previous similar messages [ 5443.583926] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5443.647204] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 5446.285486] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5478.702958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.656206] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5480.661841] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473 [ 5482.182621] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5489.574388] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5490.924059] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5532.426636] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:21:44 (1776183704) [ 5554.056637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.572223] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5569.546274] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5281 [ 5569.548766] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505 [ 5598.937566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5608.421818] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 5608.437554] Lustre: Skipped 28 previous similar messages [ 5608.615392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.618675] Lustre: Skipped 6 previous similar messages [ 5612.307837] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5614.101370] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5313 [ 5614.108370] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537 [ 5620.662734] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5622.108710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5628.783723] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:23:20 (1776183800) [ 5658.335116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5660.496924] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193 [ 5660.499915] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569 [ 5662.008266] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5669.542576] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5670.759916] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5678.586388] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:24:10 (1776183850) [ 5688.600191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.687438] Lustre: *** cfs_fail_loc=605, val=0*** [ 5688.692083] LustreError: 172427:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=00000000ec009c9d failed: rc = -95 [ 5688.699842] LustreError: 172427:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5688.704141] LustreError: 172427:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5688.709325] LustreError: 172427:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5688.716334] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5688.722333] LustreError: 172427:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5688.733384] LustreError: 172427:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5695.599703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.224834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5702.167259] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5315 to 0x0:5345 [ 5702.167264] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601 [ 5710.769619] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:24:41 (1776183881) [ 5718.529720] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5744.702840] LDISKFS-fs (dm-0): recovery complete [ 5744.706981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.669231] Lustre: *** cfs_fail_loc=707, val=0*** [ 5752.049871] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5792.684699] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 5793.202708] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5614 to 0x0:5633 [ 5793.204850] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5359 to 0x0:5377 [ 5798.406824] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5800.059434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5808.907765] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:26:20 (1776183980) [ 5837.333994] LustreError: 8680:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 5843.391129] LustreError: 8680:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5861.641833] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:27:13 (1776184033) [ 5892.709241] LustreError: 8251:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 5898.769176] LustreError: 8251:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 5906.202716] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:27:58 (1776184078) [ 5933.535787] LustreError: 10043:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5938.591154] LustreError: 10043:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5945.463131] LustreError: 64694:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5960.671826] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:28:52 (1776184132) [ 6016.026538] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:29:47 (1776184187) [ 6043.761876] LustreError: 6275:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6043.772144] LustreError: 6275:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 6044.199633] LustreError: 6275:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6048.036799] LustreError: 8680:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6048.046739] LustreError: 8680:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 15 previous similar messages [ 6052.223885] LustreError: 8248:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6052.234581] LustreError: 8248:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 29 previous similar messages [ 6056.157974] LustreError: 43098:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6056.177648] LustreError: 43098:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 29 previous similar messages [ 6068.567733] LustreError: 6275:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6068.576289] LustreError: 6275:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6075.876402] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:30:47 (1776184247) [ 6107.269081] Lustre: DEBUG MARKER: phase 2 [ 6114.651720] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:31:26 (1776184286) [ 6195.641520] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:32:47 (1776184367) [ 6197.137610] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6198.795343] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:32:50 (1776184370) [ 6202.953955] Lustre: DEBUG MARKER: Started rundbench load pid=138616 ... [ 6210.921052] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6213.454890] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6215.122193] Lustre: Failing over lustre-MDT0000 [ 6215.127307] Lustre: Skipped 7 previous similar messages [ 6215.326647] Lustre: server umount lustre-MDT0000 complete [ 6215.332086] Lustre: Skipped 8 previous similar messages [ 6215.592616] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.609939] LustreError: Skipped 200 previous similar messages [ 6215.648838] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6215.660996] LustreError: Skipped 15 previous similar messages [ 6215.667334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.684259] Lustre: Skipped 22 previous similar messages [ 6225.183444] Lustre: 3368:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184390/real 1776184390] req@00000000506f623c x1862457901750144/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184397 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 6225.212937] Lustre: 3368:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6225.225167] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6225.237462] LustreError: Skipped 4 previous similar messages [ 6235.877103] LDISKFS-fs (dm-0): recovery complete [ 6235.884702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.762444] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0b95b3 to 0x65248e50bb0cb355 [ 6241.774535] Lustre: Skipped 4 previous similar messages [ 6241.779264] Lustre: MGC192.168.204.131@tcp: Connection restored to (at 0@lo) [ 6241.791670] Lustre: Skipped 16 previous similar messages [ 6242.025309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.038302] Lustre: Skipped 3 previous similar messages [ 6242.086231] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6242.097820] Lustre: Skipped 6 previous similar messages [ 6243.216462] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6243.223320] Lustre: Skipped 7 previous similar messages [ 6245.322065] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6247.440493] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6247.448523] Lustre: Skipped 7 previous similar messages [ 6247.476447] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5702 to 0x0:5729 [ 6247.476501] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5427 to 0x0:5473 [ 6254.170069] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6255.579314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6265.005651] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6267.719303] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6269.618333] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 6294.402029] LDISKFS-fs (dm-1): recovery complete [ 6294.406648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.563513] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6300.919625] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:364 to 0x280000400:385 [ 6300.920377] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:171 to 0x2c0000400:193 [ 6308.367861] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6309.877743] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6319.521855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6322.030538] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6345.510493] LDISKFS-fs (dm-0): recovery complete [ 6345.515115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.808744] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6355.541313] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5702 to 0x0:5761 [ 6355.543294] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5427 to 0x0:5505 [ 6362.111857] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6363.511517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6370.831631] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:35:42 (1776184542) [ 6498.300548] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6510.609527] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6512.507605] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 6537.916802] LDISKFS-fs (dm-1): recovery complete [ 6537.924856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.519631] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6548.033705] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:601 to 0x2c0000400:641 [ 6548.033706] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:794 to 0x280000400:833 [ 6555.372288] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6556.756462] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6598.703268] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:39:30 (1776184770) [ 6728.150497] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6740.480362] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 6742.409465] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 6744.996088] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 6745.015230] Lustre: Skipped 4 previous similar messages [ 6771.424426] LDISKFS-fs (dm-1): recovery complete [ 6771.430042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.870738] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6776.262496] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6779.160145] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:927 to 0x280000400:961 [ 6779.164027] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:734 to 0x2c0000400:769 [ 6786.340380] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6788.062790] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6796.067105] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:42:47 (1776184967) [ 6925.457082] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6937.605895] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 6939.566504] Lustre: Failing over lustre-MDT0001 [ 6939.572410] Lustre: Skipped 4 previous similar messages [ 6939.634159] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 6939.641891] LustreError: Skipped 4 previous similar messages [ 6939.651493] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.662099] Lustre: Skipped 16 previous similar messages [ 6939.665952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6939.779373] Lustre: server umount lustre-MDT0001 complete [ 6939.783886] Lustre: Skipped 4 previous similar messages [ 6941.156864] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6941.176083] LustreError: Skipped 130 previous similar messages [ 6964.933533] LDISKFS-fs (dm-1): recovery complete [ 6964.941252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6965.154804] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6965.201804] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6965.210711] Lustre: Skipped 4 previous similar messages [ 6966.375950] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6966.385898] Lustre: Skipped 4 previous similar messages [ 6970.355863] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 6970.367332] Lustre: Skipped 18 previous similar messages [ 6970.405654] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6970.416937] Lustre: Skipped 4 previous similar messages [ 6970.479209] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:776 to 0x2c0000400:801 [ 6971.229157] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6982.406940] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6983.986679] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6992.508986] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:46:04 (1776185164) [ 7002.572612] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7004.942694] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7006.151170] Lustre: lustre-OST0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnecting [ 7006.164778] Lustre: Skipped 1 previous similar message [ 7030.547565] LDISKFS-fs (dm-2): recovery complete [ 7030.551314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7030.723339] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7032.019427] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6753 [ 7032.025421] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:968 to 0x280000400:993 [ 7035.738374] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7045.790064] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7047.475435] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7061.890426] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7064.488990] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7092.708143] LDISKFS-fs (dm-2): recovery complete [ 7092.711986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7092.937990] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7094.640391] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6785 [ 7094.653798] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:968 to 0x280000400:1025 [ 7099.486155] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7110.024099] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7111.676962] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7122.604274] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:48:14 (1776185294) [ 7250.596789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7259.029498] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7272.101813] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7274.625532] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 7280.398500] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776185454 with bad export cookie 7288106574379108007 [ 7280.406512] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7280.414587] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7280.433960] LustreError: Skipped 1 previous similar message [ 7289.759521] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185456/real 1776185456] req@000000004acd6ef3 x1862457904824192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776185463 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 7289.816765] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7306.586754] LDISKFS-fs (dm-0): recovery complete [ 7306.589469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.388607] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb0e46a7 to 0x65248e50bb1ea9fa [ 7313.410636] Lustre: Skipped 1 previous similar message [ 7313.755751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7313.766359] Lustre: Skipped 3 previous similar messages [ 7318.267728] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7343.833969] LDISKFS-fs (dm-1): recovery complete [ 7343.837930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.287285] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7350.576919] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:968 to 0x280000400:1057 [ 7350.577588] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:776 to 0x2c0000400:833 [ 7357.587939] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6463 to 0x0:6497 [ 7357.591925] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6817 [ 7364.273302] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7365.836808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7367.410589] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7374.792579] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 12:52:26 (1776185546) [ 7382.186752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7408.501194] LDISKFS-fs (dm-0): recovery complete [ 7408.510046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.918136] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7415.819829] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7419.330851] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 7419.386287] Lustre: 199328:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000007357a10b x1862457886321152/t330712498153(330712498153) o101->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:534/0 lens 648/3424 e 0 to 0 dl 1776185599 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7419.515242] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6849 [ 7419.521365] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6499 to 0x0:6529 [ 7425.129514] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7426.668081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7434.561588] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 12:53:26 (1776185606) [ 7441.642488] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7469.584410] LDISKFS-fs (dm-0): recovery complete [ 7469.589019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.245290] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7474.248460] LustreError: 203250:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007e0590bc x1862457886321152/t330712498153(330712498153) o101->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:588/0 lens 648/600 e 0 to 0 dl 1776185653 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7477.646531] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7481.275685] Lustre: lustre-MDT0000: Client 4f172e80-1084-4ee2-a45f-280c1b2bb9f1 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7481.298475] Lustre: 199326:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000000e13a850 x1862457886321152/t330712498153(330712498153) o101->4f172e80-1084-4ee2-a45f-280c1b2bb9f1@192.168.204.31@tcp:595/0 lens 648/3424 e 0 to 0 dl 1776185660 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7481.442520] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6561 [ 7481.446764] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6881 [ 7487.336588] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7489.024522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7497.415916] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 12:54:29 (1776185669) [ 7527.204965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.648849] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7538.211471] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6531 to 0x0:6593 [ 7543.266424] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.274369] LustreError: Skipped 193 previous similar messages [ 7546.020679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7547.908134] Lustre: lustre-OST0000: Denying connection for new client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7547.933837] Lustre: Skipped 12 previous similar messages [ 7548.135264] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6913 [ 7548.165791] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:968 to 0x280000400:1089 [ 7551.451989] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7563.993853] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 12:55:35 (1776185735) [ 7565.357090] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7565.363806] LustreError: 199332:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000017f8720c x1862457905075520/t347892350983(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:680/0 lens 1056/4320 e 0 to 0 dl 1776185745 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7571.434762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7571.472930] Lustre: Skipped 25 previous similar messages [ 7571.506026] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7571.515636] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.204.131@tcp (at 0@lo) [ 7571.534835] Lustre: Skipped 29 previous similar messages [ 7575.618220] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7577.732738] Lustre: Failing over lustre-MDT0000 [ 7577.746505] Lustre: Skipped 8 previous similar messages [ 7578.067763] Lustre: server umount lustre-MDT0000 complete [ 7578.071727] Lustre: Skipped 8 previous similar messages [ 7601.150255] LDISKFS-fs (dm-0): recovery complete [ 7601.158321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.196735] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7606.213974] Lustre: Skipped 8 previous similar messages [ 7607.778777] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7607.786549] Lustre: Skipped 8 previous similar messages [ 7611.438030] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7611.446294] Lustre: Skipped 8 previous similar messages [ 7611.512092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6945 [ 7611.512763] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6625 [ 7611.736283] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7623.030683] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7624.732813] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7634.385859] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 12:56:46 (1776185806) [ 7635.520638] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7635.524214] LustreError: 199332:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007c1cd187 x1862457905099968/t352187318282(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:750/0 lens 2200/4320 e 0 to 0 dl 1776185815 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7642.366942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7642.602918] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7649.819075] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7652.320060] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7652.333353] LustreError: Skipped 5 previous similar messages [ 7652.346674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7684.128468] LDISKFS-fs (dm-1): recovery complete [ 7684.141209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.522813] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7689.804580] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1100 to 0x280000400:1121 [ 7689.820146] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:844 to 0x2c0000400:865 [ 7699.354054] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7701.448637] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7710.373352] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 12:58:02 (1776185882) [ 7711.588797] LustreError: 209757:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000000fb4f5bb x1862457905122368/t352187318301(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:71/0 lens 2200/4320 e 0 to 0 dl 1776185891 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7718.884783] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7720.144612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7729.103955] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7757.850903] LDISKFS-fs (dm-0): recovery complete [ 7757.869260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.744972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:6977 [ 7764.748489] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6657 [ 7765.528793] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7774.563148] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7776.181532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7802.917748] LDISKFS-fs (dm-1): recovery complete [ 7802.920500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7807.526082] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7808.580853] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:876 to 0x2c0000400:897 [ 7808.582086] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1132 to 0x280000400:1153 [ 7819.301616] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7820.936723] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7830.940532] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:00:02 (1776186002) [ 7832.256590] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7832.259115] Lustre: Skipped 1 previous similar message [ 7838.691152] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7842.728696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7851.491711] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7858.337111] LustreError: 44574:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186032 with bad export cookie 7288106574380221875 [ 7858.357715] LustreError: 44574:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7883.348483] LDISKFS-fs (dm-0): recovery complete [ 7883.353505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.203992] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7921.509373] LDISKFS-fs (dm-1): recovery complete [ 7921.512844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.883083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.889126] Lustre: Skipped 10 previous similar messages [ 7926.647331] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7928.237366] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1164 to 0x280000400:1185 [ 7928.237724] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:908 to 0x2c0000400:929 [ 7928.390150] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6689 [ 7928.391155] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7009 [ 7936.366291] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7938.626482] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7940.382457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7951.423442] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:02:02 (1776186122) [ 7952.574026] LustreError: 217856:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000cadb7ba6 x1862457887287936/t38654705734(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:312/0 lens 560/448 e 0 to 0 dl 1776186132 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 7952.593261] LustreError: 217856:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 7959.955668] Lustre: lustre-MDT0001: Client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp) reconnecting [ 7959.960628] Lustre: Skipped 1 previous similar message [ 7959.971094] Lustre: 217854:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000000fb4f5bb x1862457887287936/t38654705734(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:319/0 lens 560/448 e 0 to 0 dl 1776186139 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 7963.402452] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7974.368159] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186141/real 1776186141] req@000000005095d1ee x1862457905198144/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186148 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 7974.403510] Lustre: 3370:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [ 7974.428379] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7974.452713] LustreError: Skipped 6 previous similar messages [ 7992.645836] LDISKFS-fs (dm-0): recovery complete [ 7992.650770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.165650] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb1f5526 to 0x65248e50bb1f6388 [ 8008.175080] Lustre: Skipped 6 previous similar messages [ 8012.075790] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8013.880399] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6721 [ 8013.883938] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7041 [ 8022.036077] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8023.677781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8037.689277] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:03:28 (1776186208) [ 8044.945850] Lustre: lustre-MDT0001: Client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp) reconnecting [ 8044.985041] Lustre: 217855:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000000592f55e x1862457887306816/t38654705804(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:404/0 lens 560/448 e 0 to 0 dl 1776186224 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8047.890172] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8075.382670] LDISKFS-fs (dm-1): recovery complete [ 8075.385253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.173449] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8080.903160] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:950 to 0x2c0000400:993 [ 8080.903347] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1206 to 0x280000400:1249 [ 8090.116564] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8092.228093] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8105.006278] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:04:35 (1776186275) [ 8107.158758] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8107.164920] Lustre: Skipped 2 previous similar messages [ 8114.590895] Lustre: lustre-MDT0001: Client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp) reconnecting [ 8114.612707] Lustre: 217856:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000625e3e48 x1862457887323904/t42949673032(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:474/0 lens 560/448 e 0 to 0 dl 1776186294 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8119.867321] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8128.621048] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8144.352592] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.367785] LustreError: Skipped 238 previous similar messages [ 8152.550629] LDISKFS-fs (dm-0): recovery complete [ 8152.556913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.497421] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7073 [ 8161.501856] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6753 [ 8161.765238] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8172.005684] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8173.547543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8181.217430] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.227041] Lustre: Skipped 34 previous similar messages [ 8204.883815] LDISKFS-fs (dm-1): recovery complete [ 8204.887475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.930956] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 8210.932982] Lustre: Skipped 39 previous similar messages [ 8211.074196] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1004 to 0x2c0000400:1025 [ 8211.076817] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1260 to 0x280000400:1281 [ 8213.042600] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8224.308855] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8227.286649] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8236.397369] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:06:48 (1776186408) [ 8237.361972] LustreError: 217855:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000dbdb9fec x1862457887347072/t47244640326(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:597/0 lens 560/448 e 0 to 0 dl 1776186417 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8237.386687] LustreError: 217855:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 2 previous similar messages [ 8244.690602] Lustre: lustre-MDT0001: Client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp) reconnecting [ 8244.728266] Lustre: 219586:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000b786338b x1862457887347072/t47244640326(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:604/0 lens 560/448 e 0 to 0 dl 1776186424 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8248.739899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8256.352470] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8258.502522] Lustre: Failing over lustre-MDT0000 [ 8258.505990] Lustre: Skipped 9 previous similar messages [ 8258.902409] Lustre: server umount lustre-MDT0000 complete [ 8258.913707] Lustre: Skipped 9 previous similar messages [ 8261.602884] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8261.612992] LustreError: Skipped 6 previous similar messages [ 8263.199298] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186436 with bad export cookie 7288106574380235532 [ 8263.228573] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8287.968560] LDISKFS-fs (dm-0): recovery complete [ 8287.977795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8296.467493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8296.481047] Lustre: Skipped 9 previous similar messages [ 8302.104263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8313.236604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8313.247674] Lustre: Skipped 9 previous similar messages [ 8328.103869] LDISKFS-fs (dm-1): recovery complete [ 8328.109885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.636041] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 2 recovered and 0 were evicted. [ 8334.648253] Lustre: Skipped 9 previous similar messages [ 8334.698811] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7105 [ 8334.713915] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6785 [ 8334.778300] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1036 to 0x2c0000400:1057 [ 8334.779281] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1292 to 0x280000400:1313 [ 8335.843166] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8350.896485] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8353.663711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8355.594408] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8366.238412] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:08:57 (1776186537) [ 8375.266302] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8376.331341] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8402.382469] LDISKFS-fs (dm-1): recovery complete [ 8402.384969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.131216] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8408.115440] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1089 [ 8417.704229] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8419.414319] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8428.751222] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:10:00 (1776186600) [ 8437.229162] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8437.726790] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8463.059597] LDISKFS-fs (dm-0): recovery complete [ 8463.066257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8471.628245] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8472.197929] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6817 [ 8472.198798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7137 [ 8482.239687] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8484.132449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8492.830549] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:11:04 (1776186664) [ 8501.216667] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8502.181398] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8512.170302] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8541.526424] LDISKFS-fs (dm-0): recovery complete [ 8541.532555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.710418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8556.726715] Lustre: Skipped 8 previous similar messages [ 8561.572737] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8562.312388] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7169 [ 8562.312785] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6849 [ 8571.373840] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8572.862956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8599.938659] LDISKFS-fs (dm-1): recovery complete [ 8599.943415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.739171] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8605.729380] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1121 [ 8605.731282] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1345 [ 8614.486283] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8616.227570] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8625.644360] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:13:16 (1776186796) [ 8626.871350] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8626.883238] Lustre: Skipped 4 previous similar messages [ 8633.864372] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8634.335153] Lustre: 238821:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186800/real 1776186800] req@000000009c5782bb x1862457905390976/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1728/4320 e 0 to 1 dl 1776186807 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 8634.361781] Lustre: 238821:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 8634.373789] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8641.674273] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8647.103325] LustreError: 6258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186820 with bad export cookie 7288106574380245724 [ 8647.105881] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.115033] LustreError: 6258:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8647.130897] LustreError: Skipped 4 previous similar messages [ 8672.797834] LDISKFS-fs (dm-0): recovery complete [ 8672.802250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.393403] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb1fa2dc to 0x65248e50bb1fabf0 [ 8679.404667] Lustre: Skipped 4 previous similar messages [ 8685.139819] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8711.896649] LDISKFS-fs (dm-1): recovery complete [ 8711.900216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.502335] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8717.530681] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1153 [ 8717.779797] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7201 [ 8717.785285] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6881 [ 8728.031344] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8729.862598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8731.591526] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8739.876353] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:15:11 (1776186911) [ 8748.065240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8749.466879] Lustre: lustre-MDT0001: Client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp) reconnecting [ 8749.481470] Lustre: 243229:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000093a9ca21 x1862457887407040/t64424509442(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:354/0 lens 496/2888 e 0 to 0 dl 1776186929 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 8750.016806] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.036302] LustreError: Skipped 225 previous similar messages [ 8776.111026] LDISKFS-fs (dm-0): recovery complete [ 8776.117736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.501384] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6913 [ 8782.501889] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7233 [ 8782.684208] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8794.371968] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8796.026706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8804.378140] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:16:16 (1776186976) [ 8805.739331] LustreError: 241948:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000083e5ab88 x1862457887415552/t64424509444(0) o36->abe7124e-37c5-4288-b52a-72bb04c21098@192.168.204.31@tcp:410/0 lens 496/456 e 0 to 0 dl 1776186985 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 8805.776669] LustreError: 241948:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 5 previous similar messages [ 8813.129196] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8818.155990] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8818.169931] Lustre: Skipped 38 previous similar messages [ 8837.991895] LDISKFS-fs (dm-1): recovery complete [ 8837.994941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.773728] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 8843.792934] Lustre: Skipped 41 previous similar messages [ 8843.876062] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1377 [ 8843.876439] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1185 [ 8844.075379] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8854.155660] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8855.717315] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8863.617501] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:17:15 (1776187035) [ 8870.540818] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8878.326196] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8880.362886] Lustre: Failing over lustre-MDT0000 [ 8880.371383] Lustre: Skipped 9 previous similar messages [ 8880.667179] Lustre: server umount lustre-MDT0000 complete [ 8880.672390] Lustre: Skipped 9 previous similar messages [ 8884.210241] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8884.223918] LustreError: Skipped 6 previous similar messages [ 8906.108593] LDISKFS-fs (dm-0): recovery complete [ 8906.111819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.092642] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8909.113581] Lustre: Skipped 9 previous similar messages [ 8914.209409] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8914.628655] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6945 [ 8914.635235] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7265 [ 8923.983269] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8925.597326] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8952.247901] LDISKFS-fs (dm-1): recovery complete [ 8952.252587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.916590] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8953.926098] Lustre: Skipped 10 previous similar messages [ 8956.404428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8957.961273] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8957.974957] Lustre: Skipped 10 previous similar messages [ 8958.012470] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1409 [ 8958.012568] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1217 [ 8965.920303] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8967.520820] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8976.043039] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:19:07 (1776187147) [ 8984.466602] Lustre: lustre-MDT0001: Client abe7124e-37c5-4288-b52a-72bb04c21098 (at 192.168.204.31@tcp) reconnecting [ 8984.479538] Lustre: Skipped 2 previous similar messages [ 8984.757974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8991.955112] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8997.827615] LustreError: 187105:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187171 with bad export cookie 7288106574380252906 [ 8997.839111] LustreError: 187105:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9022.764061] LDISKFS-fs (dm-0): recovery complete [ 9022.776433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.234020] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9062.673384] LDISKFS-fs (dm-1): recovery complete [ 9062.683104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.874370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9069.466706] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1249 [ 9069.469116] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1441 [ 9069.493522] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6595 to 0x0:6977 [ 9069.501698] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6719 to 0x0:7297 [ 9080.846616] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9082.813460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9084.000567] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9093.881824] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:21:05 (1776187265) [ 9095.893904] Lustre: 255371:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting abe7124e-37c5-4288-b52a-72bb04c21098 at adminstrative request [ 9105.830540] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:21:17 (1776187277) [ 9133.163038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9144.583799] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9145.054209] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7029 to 0x0:7073 [ 9145.060366] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7348 to 0x0:7393 [ 9154.800739] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9156.190944] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9164.429736] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:22:15 (1776187335) [ 9201.103660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9201.336701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.341249] Lustre: Skipped 10 previous similar messages [ 9203.152981] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7494 to 0x0:7521 [ 9203.165722] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1473 [ 9207.540473] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9218.056766] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9219.374444] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9228.747644] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:23:20 (1776187400) [ 9247.647177] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187414/real 1776187414] req@00000000d50a84da x1862457905564160/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187421 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 9247.651934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.694124] Lustre: 3371:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [ 9247.706139] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9247.726732] LustreError: Skipped 4 previous similar messages [ 9259.268149] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9259.595850] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7494 to 0x0:7553 [ 9259.596134] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7029 to 0x0:7105 [ 9269.507962] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:24:01 (1776187441) [ 9277.755695] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9302.499569] LDISKFS-fs (dm-2): recovery complete [ 9302.504805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9304.230104] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7555 to 0x0:7585 [ 9304.237519] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1505 [ 9307.446993] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9318.918523] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9320.804451] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9329.176561] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:25:01 (1776187501) [ 9337.868818] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9353.707202] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.723613] LustreError: Skipped 231 previous similar messages [ 9368.082039] LDISKFS-fs (dm-2): recovery complete [ 9368.086338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9370.053416] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.31@tcp inode [0x20002ea31:0x5:0x0] object 0x0:7586 extent [0-4194303]: client csum d13f3ca2, server csum cddc5788 [ 9370.132951] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7587 to 0x0:7617 [ 9370.144047] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1537 [ 9373.729393] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9385.262239] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9387.266983] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9396.420835] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:26:07 (1776187567) [ 9403.176462] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9409.617892] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9418.211601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.227090] Lustre: Skipped 25 previous similar messages [ 9419.595301] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187593 with bad export cookie 7288106574380279597 [ 9419.630930] LustreError: 6260:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9444.171147] LDISKFS-fs (dm-0): recovery complete [ 9444.175318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9452.519966] Lustre: Evicted from MGS (at 192.168.204.131@tcp) after server handle changed from 0x65248e50bb20272d to 0x65248e50bb203940 [ 9452.544450] Lustre: Skipped 5 previous similar messages [ 9452.553244] Lustre: MGC192.168.204.131@tcp: Connection restored to 192.168.204.131@tcp (at 0@lo) [ 9452.563886] Lustre: Skipped 33 previous similar messages [ 9457.070447] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9458.921546] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7029 to 0x0:7137 [ 9479.501790] LDISKFS-fs (dm-2): recovery complete [ 9479.511684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9481.806803] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1569 [ 9483.960638] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9497.940855] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:27:49 (1776187669) [ 9509.473062] Lustre: Failing over lustre-OST0000 [ 9509.480156] Lustre: Skipped 10 previous similar messages [ 9509.560957] Lustre: server umount lustre-OST0000 complete [ 9509.566118] Lustre: Skipped 10 previous similar messages [ 9533.943038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9540.363757] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9540.375878] Lustre: Skipped 10 previous similar messages [ 9544.662333] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9545.802729] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7029 to 0x0:7169 [ 9551.607185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.003556] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9558.130703] Lustre: lustre-OST0000: Denying connection for new client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 9563.554972] Lustre: lustre-OST0000: Denying connection for new client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 9568.656831] Lustre: lustre-OST0000: Denying connection for new client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 9573.797400] Lustre: lustre-OST0000: Denying connection for new client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 9584.019351] Lustre: lustre-OST0000: Denying connection for new client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 9584.041301] Lustre: Skipped 1 previous similar message [ 9604.500265] Lustre: lustre-OST0000: Denying connection for new client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 9604.513417] Lustre: Skipped 3 previous similar messages [ 9623.000203] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9623.004583] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9623.027250] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 9623.035264] Lustre: Skipped 10 previous similar messages [ 9623.049692] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7668 to 0x0:7689 [ 9623.070849] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1601 [ 9627.296772] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec [ 9636.690155] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 9644.178321] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:30:15 (1776187815) [ 9648.607932] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9648.617226] LustreError: Skipped 7 previous similar messages [ 9667.868261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9669.760616] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9669.787862] Lustre: Skipped 11 previous similar messages [ 9670.186390] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7692 to 0x0:7721 [ 9670.204498] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1633 [ 9672.826431] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9683.639992] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:30:55 (1776187855) [ 9707.263963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9709.462920] LustreError: 272037:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9709.474962] LustreError: 272037:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 53 previous similar messages [ 9710.118139] Lustre: *** cfs_fail_loc=715, val=0*** [ 9711.138974] Lustre: *** cfs_fail_loc=715, val=0*** [ 9711.144660] Lustre: Skipped 1 previous similar message [ 9711.962253] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9716.193982] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:58 [ 9716.212960] Lustre: Skipped 1 previous similar message [ 9717.217076] Lustre: *** cfs_fail_loc=715, val=0*** [ 9722.336142] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:52 [ 9722.346445] Lustre: Skipped 2 previous similar messages [ 9723.362410] Lustre: *** cfs_fail_loc=715, val=0*** [ 9723.363843] Lustre: Skipped 2 previous similar messages [ 9723.878898] Lustre: lustre-OST0000: Client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp) reconnected, waiting for 3 clients in recovery for 0:51 [ 9729.504547] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:45 [ 9729.521327] Lustre: Skipped 1 previous similar message [ 9730.527174] Lustre: *** cfs_fail_loc=715, val=0*** [ 9730.529912] Lustre: Skipped 2 previous similar messages [ 9736.673373] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:38 [ 9736.690956] Lustre: Skipped 2 previous similar messages [ 9739.167206] Lustre: *** cfs_fail_loc=715, val=0*** [ 9739.172827] Lustre: Skipped 4 previous similar messages [ 9745.299261] Lustre: lustre-OST0000: Client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp) reconnected, waiting for 3 clients in recovery for 0:29 [ 9745.318792] Lustre: Skipped 3 previous similar messages [ 9749.495490] LustreError: 272037:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9749.498752] LustreError: 272037:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 9749.558971] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1665 [ 9749.562832] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7723 to 0x0:7753 [ 9759.732748] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9762.002803] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9771.914952] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:32:22 (1776187942) [ 9776.096492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.101495] Lustre: Skipped 3 previous similar messages [ 9796.915927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.977796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9803.981635] Lustre: Skipped 9 previous similar messages [ 9805.215378] Lustre: *** cfs_fail_loc=715, val=0*** [ 9805.221279] Lustre: Skipped 3 previous similar messages [ 9808.322286] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9809.405156] LustreError: 273603:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9811.367569] Lustre: lustre-MDT0000: Client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 9816.543752] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9830.879819] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9830.887236] Lustre: Skipped 1 previous similar message [ 9839.071270] Lustre: *** cfs_fail_loc=715, val=0*** [ 9839.076160] Lustre: Skipped 8 previous similar messages [ 9846.174525] Lustre: lustre-MDT0000: Client d3b47c2d-151e-481a-a6eb-79b40edb2784 (at 192.168.204.31@tcp) reconnected, waiting for 2 clients in recovery for 0:23 [ 9846.192883] Lustre: Skipped 4 previous similar messages [ 9851.360605] Lustre: 3367:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776188018/real 1776188018] req@000000008f6a0ed4 x1862457905731712/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1776188025 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 9851.392802] Lustre: 3367:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 9851.402764] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9851.408643] Lustre: Skipped 2 previous similar messages [ 9874.834627] Lustre: lustre-MDT0000: Recovery already passed deadline 0:04. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9882.000212] Lustre: lustre-MDT0000: Recovery already passed deadline 0:12. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9887.202766] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9887.224497] Lustre: Skipped 4 previous similar messages [ 9889.170379] Lustre: lustre-MDT0000: Recovery already passed deadline 0:19. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 9889.487101] LustreError: 273603:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9889.516472] Lustre: 273603:0:(ldlm_lib.c:2849:target_recovery_thread()) too long recovery - read logs [ 9889.530695] LustreError: dumping log to /tmp/lustre-log.1776188063.273603 [ 9889.687406] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7764 to 0x0:7785 [ 9889.688144] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7182 to 0x0:7201 [ 9896.350680] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9897.841904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9906.612742] Lustre: DEBUG MARKER: == replay-single test complete, duration 9604 sec ======== 13:34:38 (1776188078) [ 9928.159385] LustreError: 166-1: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9928.171544] LustreError: Skipped 3 previous similar messages [ 9940.317599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9948.778812] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9950.248015] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7764 to 0x0:7817 [ 9950.248020] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7182 to 0x0:7233 [ 9957.215695] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9958.635173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9965.566842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9965.582471] Lustre: Skipped 3 previous similar messages [ 9975.789028] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.817650] LustreError: Skipped 246 previous similar messages [ 9980.744075] LustreError: 6259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188154 with bad export cookie 7288106574380312469 [ 9980.756185] LustreError: 6259:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9980.898434] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9980.911865] Lustre: Skipped 5 previous similar messages [10014.019669] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [10016.124924] Key type lgssc unregistered [10016.334708] LNet: 278444:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10016.341697] LNet: Removed LNI 192.168.204.131@tcp [10016.974772] Key type .llcrypt unregistered [10016.978256] Key type ._llcrypt unregistered