[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 744296881 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001193] APIC: Switch to symmetric I/O mode setup [ 0.003098] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008488] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011154] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.015003] *** VALIDATE selinux *** [ 0.023632] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032041] *** VALIDATE tmpfs *** [ 0.033525] *** VALIDATE proc *** [ 0.034254] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036475] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039036] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043850] debug: unmapping init [mem 0xffffffff8a459000-0xffffffff8a460fff] [ 0.045205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046811] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054346] rcu: Hierarchical SRCU implementation. [ 0.056868] smp: Bringing up secondary CPUs ... [ 0.057918] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.065011] smp: Brought up 1 node, 4 CPUs [ 0.067017] smpboot: Max logical packages: 1 [ 0.068024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101169] node 0 deferred pages initialised in 30ms [ 0.108137] devtmpfs: initialized [ 0.109267] x86/mm: Memory block size: 128MB [ 0.114054] gcov: version magic: 0x41383552 [ 0.118098] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120379] pinctrl core: initialized pinctrl subsystem [ 0.122300] [ 0.123000] ************************************************************* [ 0.123022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124024] ** ** [ 0.125012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126024] ** ** [ 0.127014] ** This means that this kernel is built to expose internal ** [ 0.128021] ** IOMMU data structures, which may compromise security on ** [ 0.129015] ** your system. ** [ 0.130013] ** ** [ 0.131019] ** If you see this message and you are not debugging the ** [ 0.132027] ** kernel, report this immediately to your vendor! ** [ 0.133016] ** ** [ 0.134024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135024] ************************************************************* [ 0.137589] NET: Registered protocol family 16 [ 0.138927] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.139103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.140087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.142890] cpuidle: using governor menu [ 0.148363] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.154739] PCI: Using configuration type 1 for base access [ 0.159308] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.183385] cryptd: max_cpu_qlen set to 1000 [ 0.189389] ACPI: Added _OSI(Module Device) [ 0.190016] ACPI: Added _OSI(Processor Device) [ 0.191014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197547] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204811] ACPI: Interpreter enabled [ 0.207134] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210014] ACPI: Using IOAPIC for interrupt routing [ 0.214133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219779] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234097] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.238022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253798] acpiphp: Slot [2] registered [ 0.258057] acpiphp: Slot [5] registered [ 0.260253] acpiphp: Slot [6] registered [ 0.262114] acpiphp: Slot [7] registered [ 0.264112] acpiphp: Slot [8] registered [ 0.267203] acpiphp: Slot [9] registered [ 0.269212] acpiphp: Slot [10] registered [ 0.272108] acpiphp: Slot [3] registered [ 0.275146] acpiphp: Slot [4] registered [ 0.277109] acpiphp: Slot [11] registered [ 0.280124] acpiphp: Slot [12] registered [ 0.282105] acpiphp: Slot [13] registered [ 0.285120] acpiphp: Slot [14] registered [ 0.287257] acpiphp: Slot [15] registered [ 0.290518] acpiphp: Slot [16] registered [ 0.294118] acpiphp: Slot [17] registered [ 0.296295] acpiphp: Slot [18] registered [ 0.298233] acpiphp: Slot [19] registered [ 0.300102] acpiphp: Slot [20] registered [ 0.303105] acpiphp: Slot [21] registered [ 0.305110] acpiphp: Slot [22] registered [ 0.307134] acpiphp: Slot [23] registered [ 0.310354] acpiphp: Slot [24] registered [ 0.313091] acpiphp: Slot [25] registered [ 0.315104] acpiphp: Slot [26] registered [ 0.318114] acpiphp: Slot [27] registered [ 0.319203] acpiphp: Slot [28] registered [ 0.321128] acpiphp: Slot [29] registered [ 0.324308] acpiphp: Slot [30] registered [ 0.326096] acpiphp: Slot [31] registered [ 0.329087] PCI host bridge to bus 0000:00 [ 0.332022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346023] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.351380] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.358393] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361913] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377566] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.384056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.393018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.397091] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.401478] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.407239] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.415154] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.422020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.438000] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.444018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.449000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.456018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.463018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481025] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.493457] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.508017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523017] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.534000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.539017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.545016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.558015] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.569568] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577098] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595025] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.605000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630014] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.639618] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.643013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.648021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.663016] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.673000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.674452] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.676758] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.678424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.685571] iommu: Default domain type: Passthrough [ 0.690130] SCSI subsystem initialized [ 0.693195] ACPI: bus type USB registered [ 0.696136] usbcore: registered new interface driver usbfs [ 0.701089] usbcore: registered new interface driver hub [ 0.703103] usbcore: registered new device driver usb [ 0.707202] pps_core: LinuxPPS API ver. 1 registered [ 0.710012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715068] PTP clock support registered [ 0.719042] EDAC MC: Ver: 3.0.0 [ 0.721203] PCI: Using ACPI for IRQ routing [ 0.724218] NetLabel: Initializing [ 0.726012] NetLabel: domain hash size = 128 [ 0.729014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.732292] NetLabel: unlabeled traffic allowed by default [ 0.736004] vgaarb: loaded [ 0.739246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.752053] clocksource: Switched to clocksource kvm-clock [ 0.969749] VFS: Disk quotas dquot_6.6.0 [ 0.972108] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.976137] *** VALIDATE ramfs *** [ 0.978132] *** VALIDATE hugetlbfs *** [ 0.980779] pnp: PnP ACPI init [ 0.984278] pnp: PnP ACPI: found 6 devices [ 1.008095] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.011858] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.014700] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.018272] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.021852] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.025645] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.030849] NET: Registered protocol family 2 [ 1.034704] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.043340] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.049439] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.057131] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.062297] TCP: Hash tables configured (established 65536 bind 65536) [ 1.068512] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.074528] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081891] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.085879] NET: Registered protocol family 1 [ 1.091809] RPC: Registered named UNIX socket transport module. [ 1.097764] RPC: Registered udp transport module. [ 1.100062] RPC: Registered tcp transport module. [ 1.102423] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.106245] NET: Registered protocol family 44 [ 1.107822] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.110331] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.113612] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.116500] PCI: CLS 0 bytes, default 64 [ 1.119065] Unpacking initramfs... [ 3.285481] debug: unmapping init [mem 0xffffa0317cc54000-0xffffa0317ffbffff] [ 3.290392] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.297715] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.305637] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.336252] Initialise system trusted keyrings [ 5.340369] Key type blacklist registered [ 5.343400] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.374242] zbud: loaded [ 5.379869] *** VALIDATE nfs *** [ 5.382973] *** VALIDATE nfs4 *** [ 5.384839] pstore: using deflate compression [ 5.391184] Platform Keyring initialized [ 5.713161] NET: Registered protocol family 38 [ 5.718139] Key type asymmetric registered [ 5.720145] Asymmetric key parser 'x509' registered [ 5.723649] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.735925] io scheduler mq-deadline registered [ 5.737414] io scheduler kyber registered [ 5.743264] io scheduler bfq registered [ 5.754862] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.764555] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.772742] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.775939] ACPI: Power Button [PWRF] [ 6.189457] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.659510] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.400192] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.765022] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.565657] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.692315] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.753237] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.765748] Non-volatile memory driver v1.3 [ 8.775403] Linux agpgart interface v0.103 [ 8.920299] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 8.923539] vda: detected capacity change from 0 to 68222976 [ 8.987441] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.995782] vdb: detected capacity change from 0 to 1073741824 [ 9.041155] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.044539] vdc: detected capacity change from 0 to 2621440000 [ 9.089148] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.107159] vdd: detected capacity change from 0 to 2621440000 [ 9.166584] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.170585] vde: detected capacity change from 0 to 4294967296 [ 9.211146] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.217309] vdf: detected capacity change from 0 to 4294967296 [ 9.229337] libphy: Fixed MDIO Bus: probed [ 9.304510] usbcore: registered new interface driver usbserial_generic [ 9.307198] usbserial: USB Serial support registered for generic [ 9.310787] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.331935] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.349087] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.362125] mousedev: PS/2 mouse device common for all mice [ 9.369178] rtc_cmos 00:05: RTC can wake from S4 [ 9.381517] rtc_cmos 00:05: registered as rtc0 [ 9.390487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.405811] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.410210] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.429903] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.442281] intel_pstate: CPU model not supported [ 9.448873] hid: raw HID events driver (C) Jiri Kosina [ 9.457991] usbcore: registered new interface driver usbhid [ 9.460156] usbhid: USB HID core driver [ 9.480254] drop_monitor: Initializing network drop monitor service [ 9.482548] Initializing XFRM netlink socket [ 9.491983] NET: Registered protocol family 10 [ 9.494943] Segment Routing with IPv6 [ 9.499413] NET: Registered protocol family 17 [ 9.501690] mpls_gso: MPLS GSO support [ 9.528204] RAS: Correctable Errors collector initialized. [ 9.531419] AVX version of gcm_enc/dec engaged. [ 9.540816] AES CTR mode by8 optimization enabled [ 9.842226] sched_clock: Marking stable (9842206795, 0)->(11484852032, -1642645237) [ 9.854908] registered taskstats version 1 [ 9.872928] Loading compiled-in X.509 certificates [ 9.874979] zswap: loaded using pool lzo/zbud [ 10.051072] Key type big_key registered [ 10.086276] Key type encrypted registered [ 10.111745] ima: No TPM chip found, activating TPM-bypass! [ 10.118833] ima: Allocated hash algorithm: sha1 [ 10.125607] ima: No architecture policies found [ 10.127583] evm: Initialising EVM extended attributes: [ 10.133960] evm: security.selinux [ 10.153262] evm: security.ima [ 10.155288] evm: security.capability [ 10.157337] evm: HMAC attrs: 0x1 [ 10.165032] rtc_cmos 00:05: setting system clock to 2025-10-10 19:42:37 UTC (1760125357) [ 10.209970] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 10.219257] debug: unmapping init [mem 0xffffffff8a182000-0xffffffff8a458fff] [ 10.234943] Write protecting the kernel read-only data: 28672k [ 10.271473] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 10.276499] debug: unmapping init [mem 0xffffffff89114000-0xffffffff891fffff] [ 10.391433] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.443162] systemd[1]: Detected virtualization kvm. [ 10.445533] systemd[1]: Detected architecture x86-64. [ 10.447539] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 10.473416] systemd[1]: No hostname configured. [ 10.475727] systemd[1]: Set hostname to . [ 10.478402] random: systemd: uninitialized urandom read (16 bytes read) [ 10.491064] systemd[1]: Initializing machine ID from random generator. [ 10.632493] random: ln: uninitialized urandom read (6 bytes read) [ 10.914740] random: systemd: uninitialized urandom read (16 bytes read) [ 10.926714] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 10.935578] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 10.960206] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.002442] device-mapper: uevent: version 1.0.3 [ 13.004954] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 15.451661] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.665864] virtio_net virtio0 ens2: renamed from eth0 [ 16.384960] scsi host0: ata_piix [ 16.502940] scsi host1: ata_piix [ 16.505108] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 16.508018] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 20.878514] random: crng init done [ 20.880157] random: 7 urandom warning(s) missed due to ratelimiting [ 24.084241] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 26.899359] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 31.287454] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.445126] SELinux: Disabled at runtime. [ 32.607607] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.657821] systemd[1]: Detected virtualization kvm. [ 32.661943] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.375109] hrtimer: interrupt took 2497970 ns [ 34.639856] systemd[1]: initrd-switch-root.service: Succeeded. [ 34.646524] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 34.680122] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 34.686863] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 34.691683] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 34.749251] systemd[1]: Starting Journal Service... Starting Journal Service... [ 34.782838] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 35.445916] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 37.090522] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 38.560289] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 38.617268] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 39.987477] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 40.204156] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 47.389665] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 48.624305] NFS: Registering the id_resolver key type [ 48.634279] Key type id_resolver registered [ 48.641650] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 108.284124] libcfs: loading out-of-tree module taints kernel. [ 108.309182] Key type ._llcrypt registered [ 108.311683] Key type .llcrypt registered [ 108.431839] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 125.652078] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 127.301593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.321923] alg: No test for adler32 (adler32-zlib) [ 128.762164] Lustre: Lustre: Build Version: 2.16.58_106_g01a4b29 [ 129.654142] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 131.415181] Key type lgssc registered [ 133.273991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.008482] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.709222] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.327953] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.869411] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.272515] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.252124] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 199.727836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.790275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.840673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.077396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.117958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.224659] Lustre: lustre-MDT0000: new disk, initializing [ 201.307567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.323193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.490850] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.437860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.532220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 215.583167] Lustre: 6518:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.604124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.607085] Lustre: Skipped 1 previous similar message [ 215.706576] Lustre: lustre-MDT0001: new disk, initializing [ 215.790756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.832704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.850106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.877454] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.766840] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.202573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.260850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 231.467619] Lustre: lustre-OST0000: new disk, initializing [ 231.471461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.548332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.169776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.121891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 238.137887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 238.216917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.050167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 251.135289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 251.248681] Lustre: lustre-OST0001: new disk, initializing [ 251.251713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.360685] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.702788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.584485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.592096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.626081] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.383274] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 275.650857] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 285.275224] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.568895] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 292.763182] Lustre: DEBUG MARKER: Client: 2.16.58.106 [ 294.678796] Lustre: DEBUG MARKER: MDS: 2.16.58.106 [ 296.828773] Lustre: DEBUG MARKER: OSS: 2.16.58.106 [ 298.018391] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Oct 10 15:47:24 EDT 2025 [ 311.396710] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 312.990584] Lustre: DEBUG MARKER: === replay-single: start setup 15:47:39 (1760125659) === [ 316.106532] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing check_config_client /mnt/lustre [ 334.037799] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 337.384928] Lustre: 13196:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 340.458266] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 345.378662] Lustre: DEBUG MARKER: === replay-single: finish setup 15:48:11 (1760125691) === [ 346.875089] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 15:48:13 (1760125693) [ 353.518814] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 355.561529] Lustre: Failing over lustre-MDT0000 [ 355.738973] LustreError: 14070:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 355.812124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.891267] Lustre: server umount lustre-MDT0000 complete [ 360.930135] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.949040] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 361.441961] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.051995] LustreError: 6530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.069236] LustreError: 6530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 371.169203] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.186963] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 376.289845] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.309992] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 377.319227] Lustre: 3661:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760125708/real 1760125708] req@ffffa031d826a680 x1845625331227008/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760125724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 377.353721] LDISKFS-fs (dm-0): recovery complete [ 377.356542] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 377.360458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.040354] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 387.051050] LustreError: 6526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 387.551494] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031c12ace00 x1845625331234816/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 387.691890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.309766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 390.275994] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 393.187597] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 393.229262] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 396.136569] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 397.531078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 404.098326] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 15:49:10 (1760125750) [ 405.469782] Lustre: Failing over lustre-OST0000 [ 405.508062] LustreError: 15506:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 405.512941] LustreError: 15506:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 405.557715] Lustre: server umount lustre-OST0000 complete [ 407.009213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 407.019753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.030968] Lustre: Skipped 3 previous similar messages [ 407.044467] LustreError: 8411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.063952] LustreError: 8411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 408.545922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.549518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.623565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 421.760331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.667129] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 423.699935] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 423.700032] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 423.711612] Lustre: Skipped 3 previous similar messages [ 425.554348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 430.789634] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 431.748214] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 437.396788] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 15:49:44 (1760125784) [ 442.280665] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 443.658097] Lustre: Failing over lustre-MDT0000 [ 443.758526] LustreError: 17208:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 443.761609] LustreError: 17208:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 443.808981] Lustre: server umount lustre-MDT0000 complete [ 444.386209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.392405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.403929] LustreError: 7990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.415176] LustreError: 7990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 462.815340] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760125794/real 1760125794] req@ffffa030c1dd5f80 x1845625331278976/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760125810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 462.846683] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.192354] LDISKFS-fs (dm-0): recovery complete [ 463.195946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.057396] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa030c4374a80 x1845625331287424/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 473.413140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.931562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 478.690596] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 478.692638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 478.713103] Lustre: Skipped 1 previous similar message [ 478.949181] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 484.327073] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 484.347177] Lustre: Skipped 1 previous similar message [ 489.441148] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 494.561508] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 499.685179] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 509.925711] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 509.937704] Lustre: Skipped 1 previous similar message [ 530.404156] Lustre: lustre-MDT0000: Denying connection for new client c0e79025-eaf7-443b-91aa-97794301edb2 (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:17 [ 530.426846] Lustre: Skipped 3 previous similar messages [ 548.000299] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 548.005634] Lustre: 17863:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3f73437e-81cc-4e11-9f38-56895ce899ca@ [ 548.024880] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 548.038841] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 548.040976] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 548.067233] Lustre: Skipped 2 previous similar messages [ 548.124954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 548.127424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 559.179740] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 15:51:45 (1760125905) [ 565.330195] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 566.934769] Lustre: Failing over lustre-MDT0000 [ 567.055826] LustreError: 18920:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 567.105639] LustreError: 18920:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 567.156574] Lustre: server umount lustre-MDT0000 complete [ 568.802378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 568.806823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.829707] Lustre: Skipped 3 previous similar messages [ 568.846255] LustreError: 7990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.865531] LustreError: 7990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 587.231241] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760125918/real 1760125918] req@ffffa031f4d79180 x1845625331336064/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760125934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 587.254666] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.923134] LDISKFS-fs (dm-0): recovery complete [ 588.929732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 597.686382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.716498] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 603.112051] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 603.121174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 604.089745] Lustre: lustre-MDT0000: Denying connection for new client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 604.106324] Lustre: Skipped 3 previous similar messages [ 670.686834] Lustre: lustre-MDT0000: Denying connection for new client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 670.717239] Lustre: Skipped 12 previous similar messages [ 673.001941] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 673.007411] Lustre: 19572:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c0e79025-eaf7-443b-91aa-97794301edb2@ [ 673.015578] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 673.028184] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 673.032863] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 673.041440] Lustre: Skipped 2 previous similar messages [ 673.060290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 673.060298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 683.413802] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 15:53:49 (1760126029) [ 689.190724] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 690.891235] Lustre: Failing over lustre-MDT0000 [ 690.995631] LustreError: 20634:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 690.999813] LustreError: 20634:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 691.074622] Lustre: server umount lustre-MDT0000 complete [ 693.728210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 693.735485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.744300] Lustre: Skipped 3 previous similar messages [ 700.384166] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.407410] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 710.842908] LDISKFS-fs (dm-0): recovery complete [ 710.847734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.924251] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 711.106394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.151883] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 711.647775] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 713.705506] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 716.261288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 716.307335] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 716.341363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 716.341719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 719.446091] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 720.644638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 727.191690] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 15:54:33 (1760126073) [ 733.198422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 734.994418] Lustre: Failing over lustre-MDT0000 [ 735.123848] LustreError: 22459:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 735.128785] LustreError: 22459:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 735.175790] Lustre: server umount lustre-MDT0000 complete [ 736.735781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 736.736314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.748624] Lustre: Skipped 3 previous similar messages [ 753.119179] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126084/real 1760126084] req@ffffa030c32b8a80 x1845625331417984/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 753.143954] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.052497] LDISKFS-fs (dm-0): recovery complete [ 755.059830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.360862] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031e0af5180 x1845625331426176/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 763.614309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 764.653337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 766.559881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 769.009383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 769.023829] Lustre: Skipped 3 previous similar messages [ 769.097718] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 769.127354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 769.127579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 772.522825] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 773.920273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 780.727850] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 15:55:27 (1760126127) [ 786.853853] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 788.684883] Lustre: Failing over lustre-MDT0000 [ 788.829850] LustreError: 24282:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 788.833306] LustreError: 24282:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 788.893845] Lustre: server umount lustre-MDT0000 complete [ 789.471822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.478959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.499712] Lustre: Skipped 3 previous similar messages [ 804.878837] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126136/real 1760126136] req@ffffa031ff6d2680 x1845625331447680/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 804.915616] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.229302] LDISKFS-fs (dm-0): recovery complete [ 809.235904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.077220] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3c0c39 [ 815.092859] LustreError: 24911:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.201.118@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffffa031db555d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 815.300418] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 817.123212] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 818.574445] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 820.795254] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 820.822521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 820.823243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 824.518782] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 825.635875] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 831.708865] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 15:56:18 (1760126178) [ 837.869937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 839.531898] Lustre: Failing over lustre-MDT0000 [ 839.699755] LustreError: 26106:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 839.703592] LustreError: 26106:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 839.775391] Lustre: server umount lustre-MDT0000 complete [ 841.184165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 857.571190] Lustre: 3661:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126188/real 1760126188] req@ffffa030c2ed9880 x1845625331476864/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 857.591259] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 859.511426] LDISKFS-fs (dm-0): recovery complete [ 859.514869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.785277] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f4f04a80 x1845625331485184/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 867.077811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.085270] Lustre: Skipped 2 previous similar messages [ 867.133400] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 869.352347] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 870.315365] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 872.419325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 872.425647] Lustre: Skipped 8 previous similar messages [ 872.498820] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 872.534131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 872.538595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 876.155869] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.432356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 884.755497] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 15:57:11 (1760126231) [ 890.826279] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 892.724413] Lustre: Failing over lustre-MDT0000 [ 892.895818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.922128] Lustre: Skipped 7 previous similar messages [ 892.932864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.048211] Lustre: server umount lustre-MDT0000 complete [ 913.929549] LDISKFS-fs (dm-0): recovery complete [ 913.933210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.014755] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.254549] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 917.740646] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 919.618204] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 919.619292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 924.389325] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.782963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 932.990551] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 15:57:59 (1760126279) [ 933.759625] Lustre: *** cfs_fail_loc=13b, val=315*** [ 933.762263] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 933.766845] LustreError: 6527:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031ff665f80 x1845625313435264/t38654705666(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:77/0 lens 392/456 e 0 to 0 dl 1760126292 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 940.496941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 942.023468] Lustre: Failing over lustre-MDT0000 [ 942.157332] LustreError: 29807:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 942.161631] LustreError: 29807:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 942.211820] Lustre: server umount lustre-MDT0000 complete [ 956.897102] LustreError: 19937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.912483] LustreError: 19937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 145 previous similar messages [ 959.458975] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126290/real 1760126290] req@ffffa031f19cf100 x1845625331541376/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 960.942247] LDISKFS-fs (dm-0): recovery complete [ 960.946836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.671516] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031fbd78700 x1845625331549952/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 968.863392] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 970.301343] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 970.306982] Lustre: Skipped 1 previous similar message [ 971.725806] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 974.360852] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 974.366518] Lustre: Skipped 1 previous similar message [ 974.368206] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031fd870000 x1845625313435264/t38654705666(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:117/0 lens 392/456 e 0 to 0 dl 1760126332 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 974.387425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 974.387659] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 978.127354] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 979.703530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 986.295227] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 15:58:52 (1760126332) [ 991.766588] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 993.302477] Lustre: Failing over lustre-MDT0000 [ 993.510923] Lustre: server umount lustre-MDT0000 complete [ 994.783797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 994.793931] LustreError: Skipped 1 previous similar message [ 1011.168307] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126342/real 1760126342] req@ffffa031e8ccaa00 x1845625331572096/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1011.208923] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1011.214986] LustreError: Skipped 1 previous similar message [ 1013.704364] LDISKFS-fs (dm-0): recovery complete [ 1013.712838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.411923] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f19cd180 x1845625331580416/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1021.630834] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1024.637328] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1027.050310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1027.054103] Lustre: Skipped 11 previous similar messages [ 1027.135534] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1027.139412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1030.519477] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1031.896725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1039.209066] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 15:59:45 (1760126385) [ 1045.278450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1046.040352] Lustre: *** cfs_fail_loc=114, val=0*** [ 1048.078250] Lustre: Failing over lustre-MDT0000 [ 1048.275483] Lustre: server umount lustre-MDT0000 complete [ 1052.640468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.649440] Lustre: Skipped 13 previous similar messages [ 1070.177478] LDISKFS-fs (dm-0): recovery complete [ 1070.180100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.243483] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031fb900380 x1845625331613184/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1079.010223] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1084.615532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1084.623154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1087.816639] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1099.379143] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1101.988468] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1115.954952] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 16:01:01 (1760126461) [ 1126.494628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1127.803385] Lustre: *** cfs_fail_loc=128, val=0*** [ 1132.190399] Lustre: Failing over lustre-MDT0000 [ 1132.498584] LustreError: 35491:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1132.515812] LustreError: 35491:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 1132.739081] Lustre: server umount lustre-MDT0000 complete [ 1151.968434] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126482/real 1760126482] req@ffffa030c2ed8e00 x1845625331649664/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1151.999363] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1152.014675] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.020057] LustreError: Skipped 1 previous similar message [ 1157.422054] LDISKFS-fs (dm-0): recovery complete [ 1157.427036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.214112] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3c29e4 [ 1162.510979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.515964] Lustre: Skipped 4 previous similar messages [ 1162.560655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1163.241260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1163.263544] Lustre: Skipped 2 previous similar messages [ 1166.685473] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1167.909839] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1167.929691] Lustre: Skipped 2 previous similar messages [ 1167.965949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1167.966065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1174.087421] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1175.721976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1184.028497] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 16:02:10 (1760126530) [ 1190.465528] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1192.548701] Lustre: Failing over lustre-MDT0000 [ 1192.785508] Lustre: server umount lustre-MDT0000 complete [ 1193.439661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1193.448986] LustreError: Skipped 1 previous similar message [ 1214.389454] LDISKFS-fs (dm-0): recovery complete [ 1214.393684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.065462] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3c30a5 [ 1224.600840] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1226.097265] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1226.100159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1231.305873] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1232.677404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1239.875394] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 16:03:06 (1760126586) [ 1246.769207] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1248.746969] Lustre: Failing over lustre-MDT0000 [ 1249.085479] Lustre: server umount lustre-MDT0000 complete [ 1270.009365] LDISKFS-fs (dm-0): recovery complete [ 1270.013554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.930303] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031fbc20e00 x1845625331725568/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1281.913236] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1283.557960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1283.566480] Lustre: Skipped 17 previous similar messages [ 1283.756811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1283.763167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1289.463789] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.598285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1299.775102] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 16:04:05 (1760126645) [ 1307.806706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1316.807578] Lustre: Failing over lustre-MDT0000 [ 1317.054983] Lustre: server umount lustre-MDT0000 complete [ 1319.393681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.417700] Lustre: Skipped 15 previous similar messages [ 1334.756426] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126666/real 1760126666] req@ffffa031ff7e5f80 x1845625331758592/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1334.811537] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1338.705648] LDISKFS-fs (dm-0): recovery complete [ 1338.715494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.213232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1345.217659] Lustre: Skipped 2 previous similar messages [ 1350.314092] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1355.792699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1355.806744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1363.045790] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1365.830423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1403.101651] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 16:05:48 (1760126748) [ 1410.883790] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1413.286663] Lustre: Failing over lustre-MDT0000 [ 1413.454513] LustreError: 42850:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1413.466354] LustreError: 42850:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 1413.714628] Lustre: server umount lustre-MDT0000 complete [ 1433.041639] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.047873] LustreError: Skipped 3 previous similar messages [ 1436.191427] LDISKFS-fs (dm-0): recovery complete [ 1436.193532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.297410] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3cd691 [ 1443.819326] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1443.852854] Lustre: Skipped 3 previous similar messages [ 1447.785404] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1449.051413] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 1449.062118] Lustre: Skipped 3 previous similar messages [ 1449.100494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1449.105827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1455.783623] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1457.550130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1468.269753] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 16:06:54 (1760126814) [ 1476.115549] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1478.173249] Lustre: Failing over lustre-MDT0000 [ 1478.443667] Lustre: server umount lustre-MDT0000 complete [ 1479.653786] LustreError: 38710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.676903] LustreError: 38710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 255 previous similar messages [ 1502.906051] LDISKFS-fs (dm-0): recovery complete [ 1502.910741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.065517] LustreError: 45298:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1505.070699] LustreError: 45298:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffffa031f4d8ad80 x1845625331908608/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1760126852 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1505.085953] LustreError: 45298:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1505.254299] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3cdbd1 [ 1509.452712] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1510.991684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1510.994177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1517.697086] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1519.725811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1529.008167] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 16:07:54 (1760126874) [ 1537.698191] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1539.783611] Lustre: Failing over lustre-MDT0000 [ 1540.066763] Lustre: server umount lustre-MDT0000 complete [ 1541.602609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1541.616537] LustreError: Skipped 1 previous similar message [ 1564.416656] LDISKFS-fs (dm-0): recovery complete [ 1564.419888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.687569] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f1f0dc00 x1845625331946240/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1572.142807] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1572.578831] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1572.583783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1580.358960] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1582.652053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1592.275551] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 16:08:58 (1760126938) [ 1600.702503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1602.936161] Lustre: Failing over lustre-MDT0000 [ 1603.044835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.055608] Lustre: Skipped 5 previous similar messages [ 1603.279086] Lustre: server umount lustre-MDT0000 complete [ 1624.019763] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760126955/real 1760126955] req@ffffa031f4e98e00 x1845625331975680/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760126971 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1624.071749] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1626.902554] LDISKFS-fs (dm-0): recovery complete [ 1626.915253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.279530] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3ce697 [ 1634.615796] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1634.626692] Lustre: Skipped 3 previous similar messages [ 1639.302212] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1640.127850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1640.130558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1646.936874] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1649.006797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1659.961499] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 16:10:05 (1760127005) [ 1668.548159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1670.977862] Lustre: Failing over lustre-MDT0000 [ 1671.234229] Lustre: server umount lustre-MDT0000 complete [ 1698.022449] LDISKFS-fs (dm-0): recovery complete [ 1698.025564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.369546] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f1f0fb80 x1845625332024064/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1702.411662] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 12 previous similar messages [ 1702.589557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.601545] Lustre: Skipped 7 previous similar messages [ 1706.529756] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1708.182222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1708.183128] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1715.301625] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1717.178909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1726.386845] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 16:11:12 (1760127072) [ 1735.755339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1738.140925] Lustre: Failing over lustre-MDT0000 [ 1738.453440] Lustre: server umount lustre-MDT0000 complete [ 1764.065790] LDISKFS-fs (dm-0): recovery complete [ 1764.071302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.469702] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3cf117 [ 1776.278710] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1776.279956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1776.873639] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1787.124331] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1789.830876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1801.405715] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 16:12:27 (1760127147) [ 1810.171666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1812.470272] Lustre: Failing over lustre-MDT0000 [ 1812.819341] Lustre: server umount lustre-MDT0000 complete [ 1836.136170] LDISKFS-fs (dm-0): recovery complete [ 1836.142493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.595158] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1849.340893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1849.352613] Lustre: Skipped 35 previous similar messages [ 1849.572186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1849.572186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1857.086575] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1858.744840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1868.822973] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 16:13:34 (1760127214) [ 1878.575423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1881.321543] Lustre: Failing over lustre-MDT0000 [ 1881.743611] Lustre: server umount lustre-MDT0000 complete [ 1885.157065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.189017] Lustre: Skipped 31 previous similar messages [ 1906.302740] LDISKFS-fs (dm-0): recovery complete [ 1906.317874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.756230] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3cfcbd [ 1914.744606] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1916.527112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1916.527421] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1923.797120] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1925.891797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1935.269858] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 16:14:41 (1760127281) [ 1944.064399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1946.422306] Lustre: Failing over lustre-MDT0000 [ 1946.590868] LustreError: 57475:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1946.613070] LustreError: 57475:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages [ 1946.780522] Lustre: server umount lustre-MDT0000 complete [ 1963.491959] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1963.506793] LustreError: Skipped 7 previous similar messages [ 1973.269897] LDISKFS-fs (dm-0): recovery complete [ 1973.278730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.088341] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031fd79e300 x1845625332181376/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1989.122707] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 1989.618191] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1989.637288] Lustre: Skipped 7 previous similar messages [ 1994.160641] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1994.788056] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1994.794914] Lustre: Skipped 7 previous similar messages [ 1994.898874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1994.900109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 2003.275944] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2005.591835] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2016.508540] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 16:16:01 (1760127361) [ 2027.024861] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2029.861348] Lustre: Failing over lustre-MDT0000 [ 2030.253675] Lustre: server umount lustre-MDT0000 complete [ 2056.088452] LDISKFS-fs (dm-0): recovery complete [ 2056.091947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.525309] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3d0798 [ 2076.415529] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2077.272473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:897) [ 2077.272484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 2084.722534] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2086.755991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2096.217561] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 16:17:22 (1760127442) [ 2105.139523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2107.748157] Lustre: Failing over lustre-MDT0000 [ 2107.877306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.011181] Lustre: server umount lustre-MDT0000 complete [ 2108.399202] LustreError: 38710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.431756] LustreError: 38710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 335 previous similar messages [ 2132.839499] LDISKFS-fs (dm-0): recovery complete [ 2132.845766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.770869] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2145.430212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2145.433595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2153.762748] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2155.761550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2165.453387] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 16:18:31 (1760127511) [ 2174.180201] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2176.448489] Lustre: Failing over lustre-MDT0000 [ 2176.488904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.494906] LustreError: Skipped 3 previous similar messages [ 2176.497935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.503924] Lustre: Skipped 3 previous similar messages [ 2176.718125] Lustre: server umount lustre-MDT0000 complete [ 2197.478752] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760127528/real 1760127528] req@ffffa031f1f5dc00 x1845625332289408/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760127544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2197.530952] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2200.932840] LDISKFS-fs (dm-0): recovery complete [ 2200.948663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.692944] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3d1273 [ 2207.017584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2207.031833] Lustre: Skipped 7 previous similar messages [ 2211.442531] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2212.476613] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2212.477219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2220.372665] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2222.941520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2232.088686] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 16:19:38 (1760127578) [ 2240.820840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2243.175337] Lustre: Failing over lustre-MDT0000 [ 2243.438265] Lustre: server umount lustre-MDT0000 complete [ 2268.419229] LDISKFS-fs (dm-0): recovery complete [ 2268.426146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.619896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2279.629446] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 2280.148727] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2289.159909] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2291.443147] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2301.534458] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 16:20:47 (1760127647) [ 2309.452079] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2311.871234] Lustre: Failing over lustre-MDT0000 [ 2312.155683] Lustre: server umount lustre-MDT0000 complete [ 2334.620857] LDISKFS-fs (dm-0): recovery complete [ 2334.625416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.145883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.161395] Lustre: Skipped 8 previous similar messages [ 2345.410814] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2346.749795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2346.754085] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2353.746293] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2355.951069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2365.586297] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 16:21:51 (1760127711) [ 2374.273862] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2377.057191] Lustre: Failing over lustre-MDT0000 [ 2377.185148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.195665] Lustre: Skipped 3 previous similar messages [ 2377.500412] Lustre: server umount lustre-MDT0000 complete [ 2402.512327] LDISKFS-fs (dm-0): recovery complete [ 2402.545055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.512189] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2414.225109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2414.225883] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2421.447708] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2423.320640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2432.628415] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 16:22:58 (1760127778) [ 2441.150036] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2443.350598] Lustre: Failing over lustre-MDT0000 [ 2443.586946] Lustre: server umount lustre-MDT0000 complete [ 2469.870519] LDISKFS-fs (dm-0): recovery complete [ 2469.878089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.199223] LustreError: 70891:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2471.211146] LustreError: 70891:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffffa031c12a8700 x1845625332446464/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1760127818 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2471.241500] LustreError: 70891:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2476.830502] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2477.042092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2477.052817] Lustre: Skipped 39 previous similar messages [ 2477.185917] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2477.186167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2485.386725] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2487.436914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2496.818128] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 16:24:02 (1760127842) [ 2500.657287] Lustre: 71750:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 819983a9-cca1-4d39-a81c-8dc831975a3c at adminstrative request [ 2505.931536] Lustre: Failing over lustre-MDT0000 [ 2506.402944] Lustre: server umount lustre-MDT0000 complete [ 2507.745568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.773241] Lustre: Skipped 35 previous similar messages [ 2526.308069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2534.373226] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031ff58c700 x1845625332485376/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2534.430080] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 2539.894622] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2540.064872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2540.068037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2548.766140] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2550.880528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2556.759568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2565.377458] Lustre: DEBUG MARKER: before 3128, after 3128 [ 2572.416549] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 16:25:18 (1760127918) [ 2573.483520] Lustre: 73622:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 819983a9-cca1-4d39-a81c-8dc831975a3c at adminstrative request [ 2586.186112] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 16:25:31 (1760127931) [ 2595.728296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2598.157909] Lustre: Failing over lustre-MDT0000 [ 2598.325590] LustreError: 74311:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2598.336633] LustreError: 74311:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 2598.460439] Lustre: server umount lustre-MDT0000 complete [ 2617.825450] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.845234] LustreError: Skipped 8 previous similar messages [ 2623.549162] LDISKFS-fs (dm-0): recovery complete [ 2623.554631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.579195] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2628.594200] Lustre: Skipped 8 previous similar messages [ 2632.826990] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2633.832532] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2633.840933] Lustre: Skipped 8 previous similar messages [ 2633.910964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2633.922154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2641.517373] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2643.404915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2652.674386] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 16:26:38 (1760127998) [ 2661.577376] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2663.768242] Lustre: Failing over lustre-MDT0000 [ 2664.172926] Lustre: server umount lustre-MDT0000 complete [ 2687.900869] LDISKFS-fs (dm-0): recovery complete [ 2687.903813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.028522] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3d39a2 [ 2690.040245] Lustre: Skipped 1 previous similar message [ 2694.560225] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2695.810654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2695.836717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2704.186720] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2706.777323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2718.361939] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 16:27:43 (1760128063) [ 2727.232573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2729.287186] Lustre: Failing over lustre-MDT0000 [ 2729.593592] Lustre: server umount lustre-MDT0000 complete [ 2731.496250] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.531202] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 307 previous similar messages [ 2753.476389] LDISKFS-fs (dm-0): recovery complete [ 2753.496532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.855382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2762.811665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2762.814751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2770.017474] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2772.153696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2780.962138] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 16:28:47 (1760128127) [ 2788.945985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2791.151074] Lustre: Failing over lustre-MDT0000 [ 2791.450279] Lustre: server umount lustre-MDT0000 complete [ 2793.444616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2793.466515] LustreError: Skipped 4 previous similar messages [ 2808.800273] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760128140/real 1760128140] req@ffffa031fb9d8000 x1845625332632448/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760128156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2808.839530] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2815.272925] LDISKFS-fs (dm-0): recovery complete [ 2815.286347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.501137] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2819.522317] Lustre: Skipped 8 previous similar messages [ 2824.808112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2824.810173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2825.161992] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2833.829880] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2835.649650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2843.758718] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 16:29:49 (1760128189) [ 2851.216660] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2853.025156] Lustre: Failing over lustre-MDT0000 [ 2853.290269] Lustre: server umount lustre-MDT0000 complete [ 2875.629081] LDISKFS-fs (dm-0): recovery complete [ 2875.631681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.244114] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2886.718347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2886.718852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2892.733571] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2894.336792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2902.633761] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 16:30:48 (1760128248) [ 2911.177863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2913.500169] Lustre: Failing over lustre-MDT0000 [ 2913.920424] Lustre: server umount lustre-MDT0000 complete [ 2936.781046] LDISKFS-fs (dm-0): recovery complete [ 2936.784325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.313211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.324279] Lustre: Skipped 8 previous similar messages [ 2949.062969] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2949.874246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2949.876323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2957.132262] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2959.233798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2968.632272] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 16:31:54 (1760128314) [ 2975.823128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2978.083666] Lustre: Failing over lustre-MDT0000 [ 2978.398723] Lustre: server umount lustre-MDT0000 complete [ 3000.977426] LDISKFS-fs (dm-0): recovery complete [ 3000.983199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.190813] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3011.761356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3011.764623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3019.139876] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3021.158372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3030.355343] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 16:32:56 (1760128376) [ 3038.461688] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3040.453184] Lustre: Failing over lustre-MDT0000 [ 3040.687567] Lustre: server umount lustre-MDT0000 complete [ 3063.494335] LDISKFS-fs (dm-0): recovery complete [ 3063.500791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.708382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3073.603338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3073.608521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3080.284383] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3082.094377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3090.502344] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 16:33:56 (1760128436) [ 3098.195869] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3100.384100] Lustre: Failing over lustre-MDT0000 [ 3100.733468] Lustre: server umount lustre-MDT0000 complete [ 3122.415140] LDISKFS-fs (dm-0): recovery complete [ 3122.417715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.032567] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3135.462949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3135.476457] Lustre: Skipped 43 previous similar messages [ 3135.562153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3135.565239] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3141.752061] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3143.740814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3152.293723] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 16:34:58 (1760128498) [ 3159.539766] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3161.226072] Lustre: Failing over lustre-MDT0000 [ 3161.503272] Lustre: server umount lustre-MDT0000 complete [ 3166.176394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.185397] Lustre: Skipped 40 previous similar messages [ 3184.127929] LDISKFS-fs (dm-0): recovery complete [ 3184.130422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.779211] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f5240380 x1845625332860416/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3191.809418] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 3196.328096] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3197.475180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3197.477371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3203.818931] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3205.588757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3214.875857] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 16:36:00 (1760128560) [ 3222.820974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3224.871319] Lustre: Failing over lustre-MDT0000 [ 3225.029754] LustreError: 92583:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3225.033218] LustreError: 92583:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 3225.100396] Lustre: server umount lustre-MDT0000 complete [ 3244.523459] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.534602] LustreError: Skipped 9 previous similar messages [ 3247.453783] LDISKFS-fs (dm-0): recovery complete [ 3247.458505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.737633] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e3d6ca1 [ 3253.747675] Lustre: Skipped 3 previous similar messages [ 3255.783361] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3255.788654] Lustre: Skipped 9 previous similar messages [ 3258.108712] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3259.412387] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3259.420325] Lustre: Skipped 9 previous similar messages [ 3259.451519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3259.459255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3265.164765] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3266.608221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3274.133463] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 16:37:00 (1760128620) [ 3275.233140] Lustre: 93965:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 819983a9-cca1-4d39-a81c-8dc831975a3c at adminstrative request [ 3282.999991] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 16:37:09 (1760128629) [ 3289.916453] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3291.863630] Lustre: Failing over lustre-MDT0000 [ 3292.185895] Lustre: server umount lustre-MDT0000 complete [ 3303.756841] LDISKFS-fs (dm-0): recovery complete [ 3303.760725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.132894] Lustre: lustre-MDT0000: Aborting client recovery [ 3304.135580] LustreError: 95246:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3304.140278] LustreError: 95277:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -108 [ 3304.158353] Lustre: 95279:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3304.166595] Lustre: 95279:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 819983a9-cca1-4d39-a81c-8dc831975a3c@ [ 3304.174976] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3304.189996] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3304.210631] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3304.262547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1480 to 0x280000401:1505) [ 3304.279239] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 3308.226561] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3309.541813] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3330.569955] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 16:37:56 (1760128676) [ 3338.639608] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3340.362539] Lustre: Failing over lustre-MDT0000 [ 3340.544762] Lustre: server umount lustre-MDT0000 complete [ 3342.833933] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.871618] LustreError: 7991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 299 previous similar messages [ 3352.278431] LDISKFS-fs (dm-0): recovery complete [ 3352.280927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.733265] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3352.760538] Lustre: lustre-MDT0000: Aborting client recovery [ 3352.772702] LustreError: 97096:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3352.773815] Lustre: 97129:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3352.794291] Lustre: 97129:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3352.799303] Lustre: 97129:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3352.807177] Lustre: 97129:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3352.811806] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3352.824979] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3352.844748] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3352.920962] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3352.929501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3356.404604] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3358.201130] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3370.108576] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3370.111832] Lustre: Skipped 1 previous similar message [ 3377.315454] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 16:38:43 (1760128723) [ 3384.132536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3385.993552] Lustre: Failing over lustre-MDT0000 [ 3386.185556] Lustre: server umount lustre-MDT0000 complete [ 3398.948931] LDISKFS-fs (dm-0): recovery complete [ 3398.952295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.141074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3399.164187] Lustre: Skipped 4 previous similar messages [ 3399.352066] Lustre: lustre-MDT0000: Aborting client recovery [ 3399.354437] LustreError: 98947:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3399.360303] Lustre: 98980:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3399.368536] Lustre: 98980:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3399.374109] Lustre: 98980:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3399.387395] Lustre: 98980:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3399.396277] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3399.404755] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3399.422386] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3399.499829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 3399.504980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 3404.305800] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3404.779759] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3425.738755] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 16:39:31 (1760128771) [ 3426.824688] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3426.826830] LustreError: 6525:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f2117480 x1845625314535040/t201863462916(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:305/0 lens 512/456 e 0 to 0 dl 1760128785 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3430.787068] Lustre: Failing over lustre-MDT0000 [ 3431.078306] Lustre: server umount lustre-MDT0000 complete [ 3435.492704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3435.497923] LustreError: Skipped 5 previous similar messages [ 3439.384746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.839984] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3439.840529] Lustre: lustre-MDT0000: Aborting client recovery [ 3439.843173] Lustre: Skipped 16 previous similar messages [ 3439.845321] LustreError: 100258:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3439.864102] Lustre: 100291:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3439.880648] Lustre: 100291:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3439.892083] Lustre: 100291:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3439.901436] Lustre: 100291:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3439.912221] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3439.934291] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3439.958830] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3440.017318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 3440.019306] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1601) [ 3444.385421] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3445.234748] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3464.918170] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3466.640159] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 16:40:12 (1760128812) [ 3475.291268] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3478.389918] Lustre: Failing over lustre-MDT0000 [ 3480.773215] Lustre: server umount lustre-MDT0000 complete [ 3492.738423] LDISKFS-fs (dm-0): recovery complete [ 3492.741461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.041443] Lustre: lustre-MDT0000: Aborting client recovery [ 3493.043931] LustreError: 102210:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3493.048923] Lustre: 102244:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3493.061560] Lustre: 102244:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3493.069094] Lustre: 102244:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3493.078345] Lustre: 102244:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3493.086611] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3493.099474] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3493.121752] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3493.179272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 3493.179487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633) [ 3496.666392] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3498.479554] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3518.761838] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 16:41:04 (1760128864) [ 3556.621426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3558.638426] Lustre: Failing over lustre-MDT0000 [ 3558.992983] Lustre: server umount lustre-MDT0000 complete [ 3576.287330] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760128907/real 1760128907] req@ffffa030c48b3100 x1845625333159296/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760128923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3576.335873] Lustre: 3659:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 3582.612584] LDISKFS-fs (dm-0): recovery complete [ 3582.615888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.814928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.824536] Lustre: Skipped 10 previous similar messages [ 3591.096427] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3592.399363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3592.401591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3599.277331] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3601.073294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3622.887300] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 16:42:49 (1760128969) [ 3651.369898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3662.943063] Lustre: Failing over lustre-MDT0000 [ 3663.313743] Lustre: server umount lustre-MDT0000 complete [ 3686.636686] LDISKFS-fs (dm-0): recovery complete [ 3686.639520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.295849] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3699.395946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3699.396389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3702.371409] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3703.717502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3724.676487] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 16:44:30 (1760129070) [ 3726.722744] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3727.596431] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3727.606324] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3734.231286] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 16:44:40 (1760129080) [ 3761.911717] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3773.584044] Lustre: Failing over lustre-OST0000 [ 3774.954157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.976399] Lustre: Skipped 33 previous similar messages [ 3774.986438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.735332] Lustre: server umount lustre-OST0000 complete [ 3796.914658] LDISKFS-fs (dm-2): recovery complete [ 3796.919086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3798.933258] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3798.937058] Lustre: Skipped 42 previous similar messages [ 3801.372993] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3858.852657] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 16:46:45 (1760129205) [ 3865.140500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3867.521659] Lustre: Failing over lustre-MDT0000 [ 3867.646755] LustreError: 110768:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3867.651754] LustreError: 110768:0:(obd_class.h:479:obd_check_dev()) Skipped 67 previous similar messages [ 3867.748510] Lustre: server umount lustre-MDT0000 complete [ 3883.999292] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3884.011450] LustreError: Skipped 7 previous similar messages [ 3888.786703] LDISKFS-fs (dm-0): recovery complete [ 3888.793523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3894.239927] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f211fb80 x1845625333641216/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3894.276738] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 22 previous similar messages [ 3895.784762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3895.791679] Lustre: Skipped 3 previous similar messages [ 3897.718740] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3899.922897] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3899.932584] Lustre: Skipped 3 previous similar messages [ 3899.973734] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3899.973767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3904.565081] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3906.184886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3916.256223] LustreError: 111410:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3916.256667] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3917.282279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3924.155366] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 16:47:50 (1760129270) [ 3929.113660] LustreError: 38710:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3934.175479] LustreError: 38710:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3934.187141] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 3934.239267] LustreError: 8411:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 3935.761359] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3940.831917] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3940.837222] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 3942.342881] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3947.487133] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3947.497532] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 3949.045931] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3954.144828] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3955.752367] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3960.799462] LustreError: 6526:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3960.807750] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 3960.819853] Lustre: Skipped 1 previous similar message [ 3968.824333] LustreError: 6525:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3968.833371] LustreError: 6525:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3974.111175] LustreError: 6525:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3974.120530] LustreError: 6525:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3981.279335] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 3981.288504] Lustre: Skipped 2 previous similar messages [ 3989.522320] LustreError: 6524:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3989.526475] LustreError: 6524:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3994.593793] LustreError: 6524:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3994.601372] LustreError: 6524:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4002.540418] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 16:49:08 (1760129348) [ 4004.199354] LustreError: 7991:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4013.537758] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4018.656765] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4023.782717] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4026.354748] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4034.015790] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4034.027576] Lustre: Skipped 1 previous similar message [ 4044.223246] LustreError: 7991:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4044.230250] Lustre: 7991:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffffa031c4e90000 x1845625317192192/t0(0) o38->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:0/0 lens 520/416 e 0 to 0 dl 1760129371 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4044.257697] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 4044.263126] Lustre: Skipped 3 previous similar messages [ 4044.265624] LustreError: 38710:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4068.832581] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4068.844397] Lustre: Skipped 1 previous similar message [ 4084.303292] LustreError: 38710:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4084.317975] Lustre: 38710:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffffa031f4d88380 x1845625317196160/t0(0) o38->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:0/0 lens 520/416 e 0 to 0 dl 1760129411 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4089.316044] LustreError: 19937:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4113.893533] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4113.899032] Lustre: Skipped 4 previous similar messages [ 4129.327208] LustreError: 19937:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4129.337278] Lustre: 19937:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffffa031ff7da300 x1845625317199744/t0(0) o38->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:0/0 lens 520/416 e 0 to 0 dl 1760129456 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4134.369515] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 4134.378658] Lustre: Skipped 1 previous similar message [ 4134.383212] LustreError: 6525:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4158.944079] Lustre: lustre-MDT0000: Export ffffa031c55b8000 already connecting from 192.168.201.18@tcp [ 4158.952608] Lustre: Skipped 4 previous similar messages [ 4174.455355] LustreError: 6525:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4174.463604] Lustre: 6525:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffffa031ff6b9880 x1845625317203328/t0(0) o38->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:0/0 lens 520/416 e 0 to 0 dl 1760129501 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4179.427681] LustreError: 6524:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4219.471348] LustreError: 6524:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4219.478236] Lustre: 6524:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffffa031d2e7b800 x1845625317206912/t0(0) o38->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:0/0 lens 520/416 e 0 to 0 dl 1760129546 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4224.479651] LustreError: 19937:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4225.551260] LustreError: 19937:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 4229.293567] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 16:52:55 (1760129575) [ 4234.813450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4237.935904] Lustre: Failing over lustre-MDT0000 [ 4238.108275] Lustre: server umount lustre-MDT0000 complete [ 4238.822310] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.838224] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 155 previous similar messages [ 4247.122626] LDISKFS-fs (dm-0): recovery complete [ 4247.126091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.327834] Lustre: *** cfs_fail_loc=712, val=0*** [ 4247.336581] LustreError: 42156:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffffa031f1f0d500 x1845625333828352/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4247.389688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.396419] Lustre: Skipped 3 previous similar messages [ 4247.428968] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4247.429319] Lustre: lustre-MDT0000: Aborting client recovery [ 4247.433377] Lustre: Skipped 9 previous similar messages [ 4247.436225] LustreError: 116144:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4247.444816] Lustre: 116177:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4247.451739] Lustre: 116177:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 4247.457728] Lustre: 116177:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4247.465085] Lustre: 116177:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4247.473844] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4247.480180] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4247.501273] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4247.550233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4247.550234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4250.514129] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4252.643270] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4264.387827] Lustre: Failing over lustre-MDT0000 [ 4264.585279] Lustre: server umount lustre-MDT0000 complete [ 4268.000417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4268.011078] LustreError: Skipped 7 previous similar messages [ 4280.537521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.066807] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4285.958887] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4285.959220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4290.461981] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4291.754437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4298.516284] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 16:54:04 (1760129644) [ 4298.592528] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 4298.601728] Lustre: Skipped 2 previous similar messages [ 4304.542795] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 16:54:10 (1760129650) [ 4305.256874] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4305.260224] LustreError: 6534:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031ff58e300 x1845625317267712/t0(0) o700->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:428/0 lens 264/248 e 0 to 0 dl 1760129663 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4323.288259] Lustre: Failing over lustre-MDT0000 [ 4323.479408] Lustre: server umount lustre-MDT0000 complete [ 4339.673552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.690407] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4345.345635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4345.346348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4348.642792] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4349.955257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4357.579940] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 16:55:03 (1760129703) [ 4359.719803] Lustre: Failing over lustre-OST0000 [ 4359.843947] Lustre: server umount lustre-OST0000 complete [ 4376.852367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4381.890923] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4388.469945] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4389.951325] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4460.360778] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 16:56:46 (1760129806) [ 4466.988723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4469.002850] Lustre: Failing over lustre-MDT0000 [ 4469.118565] LustreError: 121323:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4469.123403] LustreError: 121323:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 4469.201722] Lustre: server umount lustre-MDT0000 complete [ 4469.217895] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.226199] Lustre: Skipped 22 previous similar messages [ 4488.998205] LDISKFS-fs (dm-0): recovery complete [ 4489.001909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.123450] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4489.129363] LustreError: Skipped 3 previous similar messages [ 4492.483311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4494.830192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4494.842486] Lustre: Skipped 20 previous similar messages [ 4495.197924] Lustre: *** cfs_fail_loc=216, val=0*** [ 4495.198114] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4495.200617] LustreError: 121960:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4496.223501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4563.050673] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 16:58:29 (1760129909) [ 4564.863513] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4564.868064] Lustre: Skipped 2 previous similar messages [ 4575.371665] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 16:58:41 (1760129921) [ 4577.737510] Lustre: Failing over lustre-MDT0000 [ 4577.928668] Lustre: server umount lustre-MDT0000 complete [ 4595.179993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.216996] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4597.230372] Lustre: Skipped 4 previous similar messages [ 4598.660278] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4600.815423] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4600.822461] LustreError: 123526:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031e0af4700 x1845625317392512/t0(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:9/0 lens 328/344 e 0 to 0 dl 1760129999 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4657.121121] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 4657.160448] Lustre: 123526:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4657.190392] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 2 recovered and 0 were evicted. [ 4657.203279] Lustre: Skipped 4 previous similar messages [ 4657.245183] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4657.247330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4660.630710] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4662.041958] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4670.140664] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 17:00:16 (1760130016) [ 4672.123583] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4678.946622] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4680.479141] Lustre: Failing over lustre-MDT0000 [ 4680.688598] Lustre: server umount lustre-MDT0000 complete [ 4699.105656] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760130030/real 1760130030] req@ffffa031fb907100 x1845625334073984/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1760130046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4699.133095] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4701.179483] LDISKFS-fs (dm-0): recovery complete [ 4701.184994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.349907] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e41f9f7 [ 4709.353402] Lustre: Skipped 1 previous similar message [ 4713.195393] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4715.050686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 4715.050886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3105) [ 4719.341576] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4720.931357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4727.703837] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 17:01:14 (1760130074) [ 4728.585031] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4735.914605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4737.433334] Lustre: Failing over lustre-MDT0000 [ 4737.620964] Lustre: server umount lustre-MDT0000 complete [ 4757.584598] LDISKFS-fs (dm-0): recovery complete [ 4757.591025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.675892] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4772.885904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3169) [ 4772.886274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4776.676325] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4778.070475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4784.526926] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 17:02:11 (1760130131) [ 4785.399053] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4792.594859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4794.056446] Lustre: Failing over lustre-MDT0000 [ 4794.270056] Lustre: server umount lustre-MDT0000 complete [ 4813.962748] LDISKFS-fs (dm-0): recovery complete [ 4813.967488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.034793] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa030c4d41180 x1845625334147072/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4827.042991] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4829.725539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 4829.725661] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 4837.290282] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 17:03:03 (1760130183) [ 4839.104547] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4839.108481] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4839.112989] LustreError: 6527:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa030c4510380 x1845625317452672/t257698037777(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:207/0 lens 392/456 e 0 to 0 dl 1760130197 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4841.245438] Lustre: Failing over lustre-MDT0000 [ 4841.483835] Lustre: server umount lustre-MDT0000 complete [ 4845.024301] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.041432] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 186 previous similar messages [ 4857.796935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.089744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4858.092698] Lustre: Skipped 8 previous similar messages [ 4858.127379] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4858.131858] Lustre: Skipped 10 previous similar messages [ 4860.801499] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4863.501198] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f211d180 x1845625317452672/t257698037777(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:231/0 lens 392/456 e 0 to 0 dl 1760130221 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4863.503040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 4863.523048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3201) [ 4866.555821] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4867.801617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4874.606685] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 17:03:40 (1760130220) [ 4875.545640] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4875.552942] LustreError: 6524:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f4d88000 x1845625317467264/t261993005072(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:287/0 lens 504/448 e 0 to 0 dl 1760130277 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4882.573873] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4883.902956] Lustre: Failing over lustre-MDT0000 [ 4883.936505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4883.938677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4883.945566] LustreError: Skipped 4 previous similar messages [ 4883.954550] Lustre: Skipped 3 previous similar messages [ 4884.078833] Lustre: server umount lustre-MDT0000 complete [ 4902.930601] LDISKFS-fs (dm-0): recovery complete [ 4902.932985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.015846] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4908.554954] Lustre: 6526:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031fd87f480 x1845625317467264/t261993005072(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:320/0 lens 504/2880 e 0 to 0 dl 1760130310 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4908.568399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 4908.570477] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 4911.936676] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4913.197164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4920.098630] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 17:04:26 (1760130266) [ 4920.966966] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4920.971654] LustreError: 6526:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031ff7c0000 x1845625317483136/t266287972368(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:333/0 lens 504/448 e 0 to 0 dl 1760130323 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4922.679237] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4928.310978] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4929.756656] Lustre: Failing over lustre-MDT0000 [ 4929.918054] Lustre: server umount lustre-MDT0000 complete [ 4948.624952] LDISKFS-fs (dm-0): recovery complete [ 4948.632946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.123152] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4954.126692] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031ff7d9c00 x1845625317483392/t266287972369(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:322/0 lens 392/456 e 0 to 0 dl 1760130312 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4954.150127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3265) [ 4954.150553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 4954.151567] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4961.879469] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 17:05:08 (1760130308) [ 4962.643914] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4962.646560] Lustre: Skipped 1 previous similar message [ 4962.650145] LustreError: 6526:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f114ce00 x1845625317496832/t270582939664(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:374/0 lens 504/448 e 0 to 0 dl 1760130364 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4962.665529] LustreError: 6526:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 4964.361976] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4964.364431] Lustre: Skipped 1 previous similar message [ 4970.622206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4972.019569] Lustre: Failing over lustre-MDT0000 [ 4972.193214] Lustre: server umount lustre-MDT0000 complete [ 4990.438924] LDISKFS-fs (dm-0): recovery complete [ 4990.445604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.676927] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6e421fcf [ 5000.681506] Lustre: Skipped 1 previous similar message [ 5003.565679] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5006.338925] Lustre: 7991:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f4d7d880 x1845625317496832/t270582939664(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:418/0 lens 504/2880 e 0 to 0 dl 1760130408 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5006.345608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 5006.345856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3329) [ 5012.489296] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 17:05:59 (1760130359) [ 5013.167435] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5014.688633] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5014.694076] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5014.696971] LustreError: 6527:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031d826bb80 x1845625317510912/t274877906960(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:383/0 lens 392/456 e 0 to 0 dl 1760130373 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5020.230298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5021.489287] Lustre: Failing over lustre-MDT0000 [ 5021.649736] Lustre: server umount lustre-MDT0000 complete [ 5039.900024] LDISKFS-fs (dm-0): recovery complete [ 5039.904660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.619565] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5045.252388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 5045.252586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 5045.257573] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031fb86aa00 x1845625317510912/t274877906960(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:413/0 lens 392/456 e 0 to 0 dl 1760130403 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5052.506067] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 17:06:39 (1760130399) [ 5053.055995] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5053.063160] LustreError: 38710:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa030c4376a00 x1845625317521664/t279172874255(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:465/0 lens 664/608 e 0 to 0 dl 1760130455 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5112.799929] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnecting [ 5112.806150] Lustre: Skipped 1 previous similar message [ 5112.811295] Lustre: 7991:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f5242300 x1845625317521664/t279172874255(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:525/0 lens 664/3488 e 0 to 0 dl 1760130515 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5116.508657] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 17:07:43 (1760130463) [ 5120.791346] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5121.838633] Lustre: Failing over lustre-MDT0000 [ 5121.933863] LustreError: 139515:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5121.937617] LustreError: 139515:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 5121.994792] Lustre: server umount lustre-MDT0000 complete [ 5122.016157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.021566] Lustre: Skipped 42 previous similar messages [ 5137.871341] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5137.877770] LustreError: Skipped 9 previous similar messages [ 5139.567029] LDISKFS-fs (dm-0): recovery complete [ 5139.571274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.926614] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5153.764416] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5153.772286] Lustre: Skipped 44 previous similar messages [ 5153.815046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 5153.815045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 5156.207630] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5157.153975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5172.892258] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 17:08:39 (1760130519) [ 5178.348406] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5179.510175] Lustre: Failing over lustre-MDT0000 [ 5179.748573] Lustre: server umount lustre-MDT0000 complete [ 5196.179668] LDISKFS-fs (dm-0): recovery complete [ 5196.182090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.417116] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5199.839725] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5199.843644] Lustre: Skipped 9 previous similar messages [ 5201.454090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3425) [ 5201.454261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 5203.267333] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5204.115144] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5207.019405] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5213.704540] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 17:09:20 (1760130560) [ 5237.016716] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5237.977785] Lustre: Failing over lustre-MDT0000 [ 5238.220837] Lustre: server umount lustre-MDT0000 complete [ 5254.645147] LDISKFS-fs (dm-0): recovery complete [ 5254.647603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.767557] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5260.348379] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5260.352215] Lustre: Skipped 10 previous similar messages [ 5260.375330] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5260.375771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5261.938620] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5262.627522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5285.531703] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 17:10:32 (1760130632) [ 5289.751361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5290.675629] Lustre: Failing over lustre-MDT0000 [ 5290.858136] Lustre: server umount lustre-MDT0000 complete [ 5306.591324] LDISKFS-fs (dm-0): recovery complete [ 5306.594198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.646608] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5312.023665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 5312.024437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 5313.782550] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5314.582121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5318.492723] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5319.359760] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5322.742044] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 17:11:09 (1760130669) [ 5328.908592] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5390.533210] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5390.535547] Lustre: Skipped 1 previous similar message [ 5390.537262] LustreError: 6525:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f5241180 x1845625320313856/t296352743435(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:47/0 lens 66040/440 e 0 to 0 dl 1760130792 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5450.721543] Lustre: 6526:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031d9474050 x1845625320313856/t296352743435(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:108/0 lens 66040/440 e 0 to 0 dl 1760130853 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5455.015317] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5455.872948] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 17:13:22 (1760130802) [ 5462.523758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5464.068080] Lustre: Failing over lustre-MDT0000 [ 5464.256723] Lustre: server umount lustre-MDT0000 complete [ 5465.572374] LustreError: 38710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.583374] LustreError: 38710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 182 previous similar messages [ 5480.677202] LDISKFS-fs (dm-0): recovery complete [ 5480.679318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.855424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5480.858267] Lustre: Skipped 8 previous similar messages [ 5480.889671] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5480.893504] Lustre: Skipped 8 previous similar messages [ 5482.790335] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5486.454711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5486.455065] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5488.286234] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5489.119247] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5495.617670] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 17:14:02 (1760130842) [ 5506.117226] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5512.200585] Lustre: Failing over lustre-OST0000 [ 5512.247344] Lustre: server umount lustre-OST0000 complete [ 5513.210132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 5513.216954] LustreError: Skipped 7 previous similar messages [ 5528.631670] LDISKFS-fs (dm-2): recovery complete [ 5528.633734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5531.475596] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5543.646273] Lustre: Failing over lustre-OST0000 [ 5543.695343] Lustre: server umount lustre-OST0000 complete [ 5557.777694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5560.795426] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5564.956588] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5565.779358] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5601.186906] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 17:15:48 (1760130948) [ 5602.538201] Lustre: Failing over lustre-MDT0000 [ 5602.862470] Lustre: server umount lustre-MDT0000 complete [ 5603.810539] LustreError: 7451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760130951 with bad export cookie 16548930913745399361 [ 5616.699722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.576306] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5622.273360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5622.273360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5630.542345] Lustre: Failing over lustre-MDT0000 [ 5630.739630] Lustre: server umount lustre-MDT0000 complete [ 5644.737578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.853486] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5650.427800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5650.427835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5651.798864] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5652.448443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5656.534934] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 17:16:43 (1760131003) [ 5668.453305] Lustre: Failing over lustre-OST0000 [ 5668.501366] Lustre: server umount lustre-OST0000 complete [ 5682.249581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5684.857063] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5688.490925] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5689.136303] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5693.529384] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 17:17:20 (1760131040) [ 5694.344229] Lustre: Failing over lustre-MDT0000 [ 5694.611924] Lustre: server umount lustre-MDT0000 complete [ 5698.648721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.690495] Lustre: *** cfs_fail_loc=605, val=0*** [ 5698.692972] LustreError: 157332:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc09cef80 failed: rc = -95 [ 5698.698809] LustreError: 157332:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 5698.702225] LustreError: 157332:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 5698.704952] LustreError: 157332:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 5698.707356] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5698.709937] LustreError: 157332:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 5698.715069] Lustre: server umount lustre-MDT0000 complete [ 5698.717202] LustreError: 157332:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5701.946769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.864207] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5707.260727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5707.260727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5707.368075] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 17:17:34 (1760131054) [ 5710.529521] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5711.916574] Lustre: Failing over lustre-MDT0000 [ 5712.291054] Lustre: server umount lustre-MDT0000 complete [ 5727.433444] LDISKFS-fs (dm-0): recovery complete [ 5727.435529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.328347] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5732.321226] Lustre: *** cfs_fail_loc=707, val=0*** [ 5791.720218] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 5791.734064] Lustre: 159487:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5791.739072] Lustre: 159487:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 5791.837374] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5791.841046] Lustre: Skipped 40 previous similar messages [ 5791.855825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 5791.855851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 5793.319348] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5793.978309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5798.406203] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 17:19:05 (1760131145) [ 5822.447620] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031ff58df80 x1845625321196544/t0(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:435/0 lens 664/0 e 0 to 0 dl 1760131180 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5822.456024] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5833.479423] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5833.488527] LustreError: 16533:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031f1fc2a00 x1845625321197824/t0(0) o35->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:446/0 lens 392/0 e 0 to 0 dl 1760131191 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5838.817345] LustreError: 42153:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031fb8f4000 x1845625335525504/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:452/0 lens 544/0 e 0 to 0 dl 1760131197 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5838.824097] LustreError: 42153:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 5846.036919] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 17:19:53 (1760131193) [ 5870.001231] LustreError: 41090:0:(tgt_handler.c:2784:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5881.031159] LustreError: 41090:0:(tgt_handler.c:2784:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5884.943942] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 17:20:31 (1760131231) [ 5908.552119] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031f181ce00 x1845625321220608/t0(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:521/0 lens 576/0 e 0 to 0 dl 1760131266 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5908.559111] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5913.655330] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5913.661147] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa030c4e95c00 x1845625321221376/t0(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:526/0 lens 664/0 e 0 to 0 dl 1760131271 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5924.719066] LustreError: 6525:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5924.735279] LustreError: 19937:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031f23d0700 x1845625321242880/t0(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:578/0 lens 664/0 e 0 to 0 dl 1760131323 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5924.747075] LustreError: 19937:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5937.648175] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 17:21:24 (1760131284) [ 6023.798269] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 17:22:50 (1760131370) [ 6047.149374] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031d8268000 x1845625321297024/t0(0) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:660/0 lens 576/0 e 0 to 0 dl 1760131405 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6047.157142] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 6047.159477] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6047.162178] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 6047.583122] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6063.475490] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6063.480108] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 6063.903162] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6063.908221] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 6079.332331] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffffa031f203d180 x1845625321315712/t0(0) o36->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:692/0 lens 504/0 e 0 to 0 dl 1760131437 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 6079.345264] LustreError: 6524:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 81 previous similar messages [ 6092.145162] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 17:23:59 (1760131439) [ 6117.309679] Lustre: DEBUG MARKER: phase 2 [ 6120.800479] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 17:24:27 (1760131467) [ 6192.288718] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 17:25:39 (1760131539) [ 6192.857692] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6193.544174] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 17:25:40 (1760131540) [ 6195.463124] Lustre: DEBUG MARKER: Started rundbench load pid=133878 ... [ 6199.674235] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6201.386615] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6202.251432] Lustre: Failing over lustre-MDT0000 [ 6202.446809] LustreError: 166384:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6202.450805] LustreError: 166384:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 6202.486431] Lustre: server umount lustre-MDT0000 complete [ 6203.367084] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6203.375922] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 90 previous similar messages [ 6203.872769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.878466] Lustre: Skipped 41 previous similar messages [ 6217.915276] LDISKFS-fs (dm-0): recovery complete [ 6217.918072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.983818] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6217.987629] LustreError: Skipped 8 previous similar messages [ 6218.094136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.099185] Lustre: Skipped 7 previous similar messages [ 6218.118672] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6218.122953] Lustre: Skipped 7 previous similar messages [ 6218.718692] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6218.723643] Lustre: Skipped 10 previous similar messages [ 6219.889015] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6223.344160] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6223.348766] Lustre: Skipped 9 previous similar messages [ 6223.365599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 6223.365646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 6224.839155] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6225.572831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6231.209464] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6232.865822] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6233.687125] Lustre: Failing over lustre-MDT0001 [ 6233.692130] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 6233.901628] Lustre: server umount lustre-MDT0001 complete [ 6238.688188] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6238.692387] LustreError: Skipped 11 previous similar messages [ 6249.244595] LDISKFS-fs (dm-1): recovery complete [ 6249.246274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6250.995102] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6254.931943] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:693 to 0x2c0000400:737) [ 6254.932085] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:710 to 0x280000400:737) [ 6256.578854] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6257.313143] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6263.060914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6264.764610] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6265.567344] Lustre: Failing over lustre-MDT0000 [ 6265.858569] Lustre: server umount lustre-MDT0000 complete [ 6281.265287] LDISKFS-fs (dm-0): recovery complete [ 6281.266791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.009845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6286.846247] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 6286.846250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 6288.344538] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6289.078798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6294.772523] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6296.474765] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 6297.271176] Lustre: Failing over lustre-MDT0001 [ 6297.278813] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 6297.489056] Lustre: server umount lustre-MDT0001 complete [ 6312.349252] LDISKFS-fs (dm-1): recovery complete [ 6312.350701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6312.416310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6312.417929] Lustre: Skipped 2 previous similar messages [ 6314.014523] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6317.878928] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1033 to 0x280000400:1057) [ 6317.878940] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1034 to 0x2c0000400:1057) [ 6319.230161] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6319.940169] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6342.186764] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 17:28:09 (1760131689) [ 6465.806572] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6476.657822] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6477.425676] Lustre: Failing over lustre-MDT0001 [ 6477.438680] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 6477.758596] Lustre: server umount lustre-MDT0001 complete [ 6492.849586] LDISKFS-fs (dm-1): recovery complete [ 6492.851680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6494.562846] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6498.273409] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6498.276541] Lustre: Skipped 14 previous similar messages [ 6500.529542] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2255 to 0x280000400:2273) [ 6500.529560] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2255 to 0x2c0000400:2273) [ 6501.983725] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6502.677708] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6627.475254] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6638.363741] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 6639.086685] Lustre: Failing over lustre-MDT0001 [ 6639.118971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6644.826810] Lustre: server umount lustre-MDT0001 complete [ 6659.555365] LDISKFS-fs (dm-1): recovery complete [ 6659.556940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6661.188581] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6667.089980] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3361 to 0x280000400:3393) [ 6667.090383] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3361 to 0x2c0000400:3393) [ 6668.524695] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6669.205769] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6685.238671] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 17:33:52 (1760132032) [ 6808.356710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6819.185650] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6819.834547] Lustre: Failing over lustre-MDT0000 [ 6819.855655] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 6819.857854] Lustre: Skipped 4 previous similar messages [ 6820.069299] LustreError: 177003:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6820.071179] LustreError: 177003:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6820.123569] Lustre: server umount lustre-MDT0000 complete [ 6823.392774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.398452] Lustre: Skipped 16 previous similar messages [ 6823.400644] LustreError: 6530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.407534] LustreError: 6530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 74 previous similar messages [ 6833.942744] LDISKFS-fs (dm-0): recovery complete [ 6833.944591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.980213] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6833.984492] LustreError: Skipped 1 previous similar message [ 6834.046983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.049648] Lustre: Skipped 5 previous similar messages [ 6834.060419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6834.063977] Lustre: Skipped 5 previous similar messages [ 6834.142678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6834.145540] Lustre: Skipped 5 previous similar messages [ 6835.305683] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6840.689345] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 6840.692845] Lustre: Skipped 5 previous similar messages [ 6840.712314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7573 to 0x280000401:7617) [ 6840.712340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7509 to 0x2c0000401:7553) [ 6842.220801] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6842.924818] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6968.116917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6979.071952] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 6979.877412] Lustre: Failing over lustre-MDT0000 [ 6979.878313] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -19 [ 6979.883216] LustreError: Skipped 4 previous similar messages [ 6979.895176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6980.206797] Lustre: server umount lustre-MDT0000 complete [ 6995.335497] LDISKFS-fs (dm-0): recovery complete [ 6995.338525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.066822] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7001.190879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7921 to 0x280000401:7937) [ 7001.191176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7856 to 0x2c0000401:7873) [ 7002.477246] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7003.122494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7006.801562] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 17:39:13 (1760132353) [ 7130.081955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7140.963873] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 7141.774384] Lustre: Failing over lustre-MDT0000 [ 7142.084720] Lustre: server umount lustre-MDT0000 complete [ 7157.140391] LDISKFS-fs (dm-0): recovery complete [ 7157.141874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.861019] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7162.337622] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7162.342070] Lustre: Skipped 13 previous similar messages [ 7165.283132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9433 to 0x280000401:9473) [ 7165.283462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9369 to 0x2c0000401:9409) [ 7166.683414] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7167.365331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7291.646890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7302.470312] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 7303.127708] Lustre: Failing over lustre-MDT0000 [ 7303.177965] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 7303.294495] Lustre: server umount lustre-MDT0000 complete [ 7317.349962] LDISKFS-fs (dm-0): recovery complete [ 7317.352121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.788594] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7325.493476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11141 to 0x280000401:11169) [ 7325.494874] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11076 to 0x2c0000401:11105) [ 7326.873627] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7327.584419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7331.399674] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 17:44:38 (1760132678) [ 7337.198995] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7338.706492] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7339.379749] Lustre: Failing over lustre-OST0000 [ 7339.416453] Lustre: server umount lustre-OST0000 complete [ 7353.647016] LDISKFS-fs (dm-2): recovery complete [ 7353.649763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7355.942757] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7359.563631] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7360.316653] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7369.419066] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7370.970217] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7371.636311] Lustre: Failing over lustre-OST0000 [ 7371.671967] Lustre: server umount lustre-OST0000 complete [ 7386.177118] LDISKFS-fs (dm-2): recovery complete [ 7386.179211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7388.340140] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7391.803461] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7392.412235] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7398.238226] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 17:45:45 (1760132745) [ 7521.181834] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7523.887565] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7534.711901] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7535.374551] Lustre: Failing over lustre-MDT0001 [ 7535.391548] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 7535.628659] LustreError: 188425:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7535.630825] LustreError: 188425:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 7535.709693] Lustre: server umount lustre-MDT0001 complete [ 7536.922268] Lustre: Failing over lustre-MDT0000 [ 7536.925739] LustreError: 42163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760132884 with bad export cookie 16548930913751338406 [ 7537.147696] Lustre: server umount lustre-MDT0000 complete [ 7551.894166] LDISKFS-fs (dm-1): recovery complete [ 7551.896460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7551.904582] LDISKFS-fs (dm-0): recovery complete [ 7551.908144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7551.950030] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7551.952419] LustreError: Skipped 3 previous similar messages [ 7552.028245] LustreError: 172749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.028552] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7552.032933] LustreError: 172749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 75 previous similar messages [ 7552.042503] Lustre: Skipped 22 previous similar messages [ 7552.059606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7552.061802] Lustre: Skipped 5 previous similar messages [ 7552.069427] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7552.071739] Lustre: Skipped 5 previous similar messages [ 7553.494428] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7553.617148] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7555.580640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7555.584491] Lustre: Skipped 5 previous similar messages [ 7556.191161] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760132887/real 1760132887] req@ffffa030cc0a5180 x1845625371093120/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760132903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7556.198219] Lustre: 3660:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 7558.249222] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7558.253913] Lustre: Skipped 5 previous similar messages [ 7558.272057] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4118 to 0x280000400:4161) [ 7558.272115] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4119 to 0x2c0000400:4161) [ 7563.299458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11226 to 0x280000401:11265) [ 7563.300502] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11163 to 0x2c0000401:11201) [ 7564.611848] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7565.306161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7565.861085] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7690.029377] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7692.600301] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7703.349473] Lustre: DEBUG MARKER: fail mds1 mds2 2 times [ 7704.003951] Lustre: Failing over lustre-MDT0000 [ 7704.201027] Lustre: server umount lustre-MDT0000 complete [ 7704.735113] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760132897/real 1760132897] req@ffffa030c174f480 x1845625371094400/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760132913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7704.746081] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7705.442846] LustreError: 42163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133052 with bad export cookie 16548930913752220266 [ 7705.443930] Lustre: Failing over lustre-MDT0001 [ 7705.446439] LustreError: 42163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7705.649493] Lustre: server umount lustre-MDT0001 complete [ 7720.477060] LDISKFS-fs (dm-1): recovery complete [ 7720.481200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7720.503029] LDISKFS-fs (dm-0): recovery complete [ 7720.504862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7720.560246] LustreError: 192967:0:(llog.c:1602:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7720.562545] Lustre: 192967:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7730.143349] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa030cc064700 x1845625381561472/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7730.148399] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 16 previous similar messages [ 7730.292709] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 7730.296547] LustreError: Skipped 4 previous similar messages [ 7731.679043] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7731.757050] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7736.376680] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4118 to 0x280000400:4193) [ 7736.376726] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4119 to 0x2c0000400:4193) [ 7741.283913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11163 to 0x2c0000401:11233) [ 7741.285263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11226 to 0x280000401:11297) [ 7742.646745] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7743.260956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7743.842202] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7747.282871] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 17:51:34 (1760133094) [ 7749.762296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7750.666869] Lustre: Failing over lustre-MDT0000 [ 7750.802326] Lustre: server umount lustre-MDT0000 complete [ 7765.007786] LDISKFS-fs (dm-0): recovery complete [ 7765.009208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7766.369969] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7768.032281] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7770.593053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7770.595753] Lustre: Skipped 23 previous similar messages [ 7784.421204] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7784.434682] Lustre: 193001:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031ed826300 x1845625391564672/t343597384260(343597384260) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:132/0 lens 520/3488 e 0 to 0 dl 1760133142 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7784.475543] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11235 to 0x2c0000401:11265) [ 7784.475545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11226 to 0x280000401:11329) [ 7785.846281] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7786.385956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7789.755397] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 17:52:16 (1760133136) [ 7792.253303] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7793.201474] Lustre: Failing over lustre-MDT0000 [ 7793.395939] Lustre: server umount lustre-MDT0000 complete [ 7807.368492] LDISKFS-fs (dm-0): recovery complete [ 7807.370474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7808.773254] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7808.999942] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7809.002444] LustreError: 192977:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa030cb158e00 x1845625391564672/t343597384260(343597384260) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:157/0 lens 520/664 e 0 to 0 dl 1760133167 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7825.382067] Lustre: lustre-MDT0000: Client 819983a9-cca1-4d39-a81c-8dc831975a3c (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [ 7825.387598] Lustre: 192978:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f2327100 x1845625391564672/t343597384260(343597384260) o101->819983a9-cca1-4d39-a81c-8dc831975a3c@192.168.201.18@tcp:173/0 lens 520/3488 e 0 to 0 dl 1760133183 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7825.417608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11297) [ 7825.417612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11226 to 0x280000401:11361) [ 7826.730283] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7827.254971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7830.640398] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 17:52:57 (1760133177) [ 7832.156426] Lustre: Failing over lustre-OST0000 [ 7832.197934] Lustre: server umount lustre-OST0000 complete [ 7833.395880] Lustre: Failing over lustre-MDT0000 [ 7833.475513] Lustre: server umount lustre-MDT0000 complete [ 7846.283021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7847.709761] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7848.451685] Lustre: lustre-MDT0000: Denying connection for new client 0f0e63dc-cfa6-47a7-af65-1463d850f7e3 (at 192.168.201.18@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7851.506718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11329) [ 7855.456385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7856.754278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11226 to 0x280000401:11393) [ 7857.492230] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7861.614683] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 17:53:28 (1760133208) [ 7861.960781] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7861.963373] LustreError: 8422:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f1b65500 x1845625383162368/t365072220167(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:210/0 lens 1312/4320 e 0 to 0 dl 1760133220 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7864.324592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7864.922238] Lustre: Failing over lustre-MDT0000 [ 7865.306322] Lustre: server umount lustre-MDT0000 complete [ 7878.111081] Lustre: 193006:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760133209/real 1760133209] req@ffffa031f1b67480 x1845625383162368/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1760133225 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7878.117475] Lustre: 193006:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 7879.335219] LDISKFS-fs (dm-0): recovery complete [ 7879.336508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.815191] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7884.791343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11425) [ 7884.791351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11361) [ 7885.877520] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7886.383664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7889.839532] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 17:53:56 (1760133236) [ 7892.705231] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7895.205579] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7895.801902] Lustre: Failing over lustre-MDT0001 [ 7895.805819] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 7902.271395] Lustre: server umount lustre-MDT0001 complete [ 7916.326613] LDISKFS-fs (dm-1): recovery complete [ 7916.329062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7917.842019] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7921.648906] Lustre: 192976:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031fc5e3100 x1845625404912512/t30064773666(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:291/0 lens 560/2880 e 0 to 0 dl 1760133301 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7921.656123] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4204 to 0x280000400:4225) [ 7921.656223] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4204 to 0x2c0000400:4225) [ 7922.806215] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7923.368887] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7926.929411] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 17:54:34 (1760133274) [ 7927.249096] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7927.251699] Lustre: Skipped 1 previous similar message [ 7927.254041] LustreError: 193729:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031fb43ce00 x1845625383223168/t369367187489(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:275/0 lens 2520/4320 e 0 to 0 dl 1760133285 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7927.259971] LustreError: 193729:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 7929.663207] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7932.157526] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7932.742868] Lustre: Failing over lustre-MDT0000 [ 7933.020657] Lustre: server umount lustre-MDT0000 complete [ 7947.017153] LDISKFS-fs (dm-0): recovery complete [ 7947.018609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.424900] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7952.396557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11457) [ 7952.396615] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11393) [ 7953.505951] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7954.041173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7955.954680] Lustre: Failing over lustre-MDT0001 [ 7956.055538] Lustre: server umount lustre-MDT0001 complete [ 7970.004318] LDISKFS-fs (dm-1): recovery complete [ 7970.005620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7971.328827] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7975.416073] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4236 to 0x280000400:4257) [ 7975.416103] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4236 to 0x2c0000400:4257) [ 7976.518526] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7977.011296] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7980.466427] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 17:55:27 (1760133327) [ 7986.335083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7988.908365] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7989.501080] Lustre: Failing over lustre-MDT0000 [ 7989.784549] Lustre: server umount lustre-MDT0000 complete [ 7990.992736] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133338 with bad export cookie 16548930913753200588 [ 7990.993805] Lustre: Failing over lustre-MDT0001 [ 7990.996071] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7996.895693] LustreError: 208649:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa030ce365f80 x1845625383284352/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7996.900549] LustreError: 208649:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7997.022726] Lustre: server umount lustre-MDT0001 complete [ 8011.892035] LDISKFS-fs (dm-1): recovery complete [ 8011.893640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8011.914452] LDISKFS-fs (dm-0): recovery complete [ 8011.918161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.975440] LustreError: 209742:0:(llog.c:1602:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8011.980574] Lustre: 209742:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8016.351396] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa030c174ea00 x1845625383285504/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8017.856028] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8017.926228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8022.583549] Lustre: 210434:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa030ce365500 x1845625404978816/t38654705732(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:394/0 lens 560/2880 e 0 to 0 dl 1760133404 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8022.590797] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4268 to 0x280000400:4289) [ 8022.592166] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4289) [ 8022.616725] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11425) [ 8022.616735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11489) [ 8023.751763] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8024.347576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8024.892483] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8028.491723] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 17:56:15 (1760133375) [ 8028.841069] LustreError: 209760:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f5075880 x1845625405012096/t42949673028(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:377/0 lens 560/448 e 0 to 0 dl 1760133387 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8028.846987] LustreError: 209760:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 8034.331834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8034.991329] Lustre: Failing over lustre-MDT0000 [ 8035.250700] Lustre: server umount lustre-MDT0000 complete [ 8045.039464] Lustre: lustre-MDT0001: Client 0f0e63dc-cfa6-47a7-af65-1463d850f7e3 (at 192.168.201.18@tcp) reconnecting [ 8045.042181] Lustre: Skipped 2 previous similar messages [ 8049.254789] LDISKFS-fs (dm-0): recovery complete [ 8049.256745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.726615] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8054.782963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11457) [ 8054.782986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11521) [ 8055.879288] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8056.425625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8060.052534] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 17:56:47 (1760133407) [ 8060.400523] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8060.401722] Lustre: Skipped 2 previous similar messages [ 8062.873277] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8063.477302] Lustre: Failing over lustre-MDT0001 [ 8063.562779] Lustre: server umount lustre-MDT0001 complete [ 8077.758084] LDISKFS-fs (dm-1): recovery complete [ 8077.760258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8079.164710] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8082.927399] Lustre: 209760:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f1fd8000 x1845625405042688/t42949673097(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:431/0 lens 560/2880 e 0 to 0 dl 1760133441 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8082.931601] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4310 to 0x280000400:4353) [ 8082.931615] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4310 to 0x2c0000400:4353) [ 8082.934355] Lustre: 209760:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8084.065518] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8084.606831] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8088.081717] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 17:57:15 (1760133435) [ 8093.807062] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8096.263144] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8096.881933] Lustre: Failing over lustre-MDT0000 [ 8097.144377] Lustre: server umount lustre-MDT0000 complete [ 8111.269023] LDISKFS-fs (dm-0): recovery complete [ 8111.270415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.663734] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8116.739652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11553) [ 8116.739661] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11489) [ 8117.928257] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8118.472761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8120.452735] Lustre: Failing over lustre-MDT0001 [ 8120.548250] Lustre: server umount lustre-MDT0001 complete [ 8134.468648] LDISKFS-fs (dm-1): recovery complete [ 8134.469848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8135.819899] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8139.767197] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4364 to 0x280000400:4385) [ 8139.767239] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4364 to 0x2c0000400:4385) [ 8140.869181] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8141.401989] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8145.028049] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 17:58:12 (1760133492) [ 8150.739341] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8153.233385] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8153.869660] Lustre: Failing over lustre-MDT0000 [ 8154.118557] LustreError: 219019:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8154.120568] LustreError: 219019:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 8154.144571] Lustre: server umount lustre-MDT0000 complete [ 8155.104274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.104721] LustreError: 209762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.110361] Lustre: Skipped 52 previous similar messages [ 8155.118127] LustreError: 209762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 183 previous similar messages [ 8155.381550] LustreError: 42158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133502 with bad export cookie 16548930913753214224 [ 8155.382559] Lustre: Failing over lustre-MDT0001 [ 8155.382870] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8155.382876] LustreError: Skipped 9 previous similar messages [ 8155.384757] LustreError: 42158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8155.520493] Lustre: server umount lustre-MDT0001 complete [ 8170.188878] LDISKFS-fs (dm-0): recovery complete [ 8170.192187] LDISKFS-fs (dm-1): recovery complete [ 8170.192385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.194054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8170.276931] LustreError: 220318:0:(llog.c:1602:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8170.279237] Lustre: 220318:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8179.679395] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031ff665880 x1845625383436800/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8179.759931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8179.763519] Lustre: Skipped 17 previous similar messages [ 8179.771392] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8179.775332] Lustre: Skipped 17 previous similar messages [ 8180.703100] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760133512/real 1760133512] req@ffffa031f1210380 x1845625383435008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1760133528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8180.710260] Lustre: 3658:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 8181.185434] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8181.214406] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8181.219668] Lustre: Skipped 18 previous similar messages [ 8181.326831] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8185.958384] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8185.961196] Lustre: Skipped 17 previous similar messages [ 8185.965664] Lustre: 220334:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f1211f80 x1845625405107840/t51539607620(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:534/0 lens 560/2880 e 0 to 0 dl 1760133544 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8185.971710] Lustre: 220334:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8185.974485] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4396 to 0x2c0000400:4417) [ 8185.974512] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4396 to 0x280000400:4417) [ 8185.991898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11585) [ 8185.991902] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11521) [ 8187.108276] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8187.621285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8188.105416] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8191.565736] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 17:58:58 (1760133538) [ 8191.924994] LustreError: 172749:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031f1211c00 x1845625383463680/t390842023950(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:540/0 lens 1744/4320 e 0 to 0 dl 1760133550 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8191.936132] LustreError: 172749:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 3 previous similar messages [ 8194.256604] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8194.889472] Lustre: Failing over lustre-MDT0001 [ 8194.893629] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 8194.897021] Lustre: Skipped 8 previous similar messages [ 8201.262524] Lustre: server umount lustre-MDT0001 complete [ 8215.097027] LDISKFS-fs (dm-1): recovery complete [ 8215.099081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8216.381571] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8220.659108] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4449) [ 8220.659116] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4449) [ 8221.881614] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8222.459508] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8225.837301] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 17:59:32 (1760133572) [ 8228.717692] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8229.350246] Lustre: Failing over lustre-MDT0000 [ 8229.633909] Lustre: server umount lustre-MDT0000 complete [ 8243.723437] LDISKFS-fs (dm-0): recovery complete [ 8243.724769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.102228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8249.347904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11617) [ 8249.347908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11553) [ 8250.424631] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8250.924232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8254.248667] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 18:00:01 (1760133601) [ 8257.108152] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8259.458204] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8260.043174] Lustre: Failing over lustre-MDT0000 [ 8260.345429] Lustre: server umount lustre-MDT0000 complete [ 8274.158068] LDISKFS-fs (dm-0): recovery complete [ 8274.160044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.498702] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8279.556286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11649) [ 8279.556323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11585) [ 8280.723252] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8281.257692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8283.151768] Lustre: Failing over lustre-MDT0001 [ 8283.246791] Lustre: server umount lustre-MDT0001 complete [ 8297.055450] LDISKFS-fs (dm-1): recovery complete [ 8297.056711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8298.362823] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8302.576154] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4481) [ 8302.576163] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4481) [ 8303.629525] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8304.116935] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8307.475276] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 18:00:54 (1760133654) [ 8310.151450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8312.708422] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8313.295986] Lustre: Failing over lustre-MDT0000 [ 8313.648313] Lustre: server umount lustre-MDT0000 complete [ 8314.754699] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133662 with bad export cookie 16548930913753224598 [ 8314.756032] Lustre: Failing over lustre-MDT0001 [ 8314.757609] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8314.759609] LustreError: 229811:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029c11:0x1f:0x0].0xf7117594 (ffffa031f20a2a00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8320.991752] LustreError: 229811:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa030cc609c00 x1845625383571200/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8320.996755] LustreError: 229811:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8321.129688] Lustre: server umount lustre-MDT0001 complete [ 8335.655372] LDISKFS-fs (dm-0): recovery complete [ 8335.656027] LDISKFS-fs (dm-1): recovery complete [ 8335.657280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8335.658444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8335.735927] LustreError: 230911:0:(llog.c:1602:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8335.738560] Lustre: 230911:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8340.448298] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6ebddf36 [ 8340.606396] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 8340.609429] LustreError: Skipped 16 previous similar messages [ 8341.975201] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8342.075261] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8345.651079] Lustre: 231667:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa031f1322680 x1845625405183360/t64424509442(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:718/0 lens 496/2888 e 0 to 0 dl 1760133728 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8345.654518] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4513) [ 8345.655277] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4513) [ 8345.664831] Lustre: 231667:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8345.678108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11681) [ 8345.678225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11617) [ 8346.866210] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8347.380200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8347.857552] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8351.177307] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 18:01:38 (1760133698) [ 8351.543572] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8351.545783] Lustre: Skipped 6 previous similar messages [ 8354.230056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8354.848391] Lustre: Failing over lustre-MDT0000 [ 8355.056371] Lustre: server umount lustre-MDT0000 complete [ 8369.113974] LDISKFS-fs (dm-0): recovery complete [ 8369.115356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.446953] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8374.241529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8374.244428] Lustre: Skipped 73 previous similar messages [ 8374.277499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11713) [ 8374.277546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11649) [ 8375.463489] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8375.980525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8407.006680] Lustre: lustre-MDT0001: Client 0f0e63dc-cfa6-47a7-af65-1463d850f7e3 (at 192.168.201.18@tcp) reconnecting [ 8407.009010] Lustre: Skipped 1 previous similar message [ 8409.299331] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 18:02:36 (1760133756) [ 8412.064764] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8412.682511] Lustre: Failing over lustre-MDT0001 [ 8412.770469] Lustre: server umount lustre-MDT0001 complete [ 8426.645408] LDISKFS-fs (dm-1): recovery complete [ 8426.647820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8428.073455] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8432.115065] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4545) [ 8432.115079] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4545) [ 8433.320675] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8433.923946] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8437.492133] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 18:03:04 (1760133784) [ 8440.202094] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8442.604779] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8443.228864] Lustre: Failing over lustre-MDT0000 [ 8443.472097] Lustre: server umount lustre-MDT0000 complete [ 8457.504459] LDISKFS-fs (dm-0): recovery complete [ 8457.506551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.894473] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8462.848932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11681) [ 8462.848938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11745) [ 8464.004493] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8464.544385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8466.436524] Lustre: Failing over lustre-MDT0001 [ 8466.524586] Lustre: server umount lustre-MDT0001 complete [ 8480.477015] LDISKFS-fs (dm-1): recovery complete [ 8480.479218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8481.951549] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8485.877366] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4577) [ 8485.877394] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4577) [ 8487.032658] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8487.583674] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8491.105380] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 18:03:58 (1760133838) [ 8491.471381] LustreError: 231667:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffffa031ff6ab800 x1845625405240576/t77309411330(0) o36->0f0e63dc-cfa6-47a7-af65-1463d850f7e3@192.168.201.18@tcp:84/0 lens 496/456 e 0 to 0 dl 1760133849 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8491.476948] LustreError: 231667:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 6 previous similar messages [ 8493.920483] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8496.419237] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8497.025170] Lustre: Failing over lustre-MDT0000 [ 8497.266151] Lustre: server umount lustre-MDT0000 complete [ 8498.490993] LustreError: 42163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133845 with bad export cookie 16548930913753232095 [ 8498.492573] Lustre: Failing over lustre-MDT0001 [ 8498.494180] LustreError: 42163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8498.640845] Lustre: server umount lustre-MDT0001 complete [ 8513.282994] LDISKFS-fs (dm-1): recovery complete [ 8513.283146] LDISKFS-fs (dm-0): recovery complete [ 8513.284500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8513.285693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.368096] LustreError: 241555:0:(llog.c:1602:llog_backup()) MGC192.168.201.118@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8513.370527] Lustre: 241555:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.118@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8523.231416] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffffa031f1320700 x1845625383715456/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8523.240866] LustreError: 3657:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 8524.787050] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8524.916388] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8529.462944] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4609) [ 8529.463057] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4609) [ 8529.478967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11267 to 0x2c0000401:11713) [ 8529.479047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11395 to 0x280000401:11777) [ 8530.714586] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8531.249388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8531.735833] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8535.069279] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 18:04:42 (1760133882) [ 8535.651698] Lustre: 242793:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0f0e63dc-cfa6-47a7-af65-1463d850f7e3 at adminstrative request [ 8540.268383] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 18:04:47 (1760133887) [ 8542.034653] Lustre: Failing over lustre-MDT0000 [ 8542.332775] Lustre: server umount lustre-MDT0000 complete [ 8554.967569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.327427] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8560.252117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11764 to 0x2c0000401:11809) [ 8560.252155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11829 to 0x280000401:11873) [ 8561.301732] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8561.818824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8565.241027] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 18:05:12 (1760133912) [ 8568.645757] Lustre: Failing over lustre-OST0000 [ 8568.713749] Lustre: server umount lustre-OST0000 complete [ 8581.266422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8583.066741] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8586.016031] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8586.510868] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8590.112104] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 18:05:37 (1760133937) [ 8591.380557] Lustre: Failing over lustre-MDT0000 [ 8591.639657] Lustre: server umount lustre-MDT0000 complete [ 8593.377580] LustreError: 42163:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760133940 with bad export cookie 16548930913753244632 [ 8594.098964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.447845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8596.127522] Lustre: lustre-MDT0000: Denying connection for new client aa1af4ab-4869-4acf-ba69-8ec3324064d9 (at 192.168.201.18@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8596.132609] Lustre: Skipped 1 previous similar message [ 8599.542437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11829 to 0x280000401:11905) [ 8599.542437] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11764 to 0x2c0000401:11841) [ 8603.676543] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 18:05:50 (1760133950) [ 8606.434272] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8607.270789] Lustre: Failing over lustre-OST0000 [ 8607.298795] Lustre: server umount lustre-OST0000 complete [ 8621.051858] LDISKFS-fs (dm-2): recovery complete [ 8621.053185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8622.837143] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8625.886272] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8626.458556] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8630.039981] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 18:06:17 (1760133977) [ 8632.868130] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8634.750927] Lustre: Failing over lustre-OST0000 [ 8634.781408] Lustre: server umount lustre-OST0000 complete [ 8648.736518] LDISKFS-fs (dm-2): recovery complete [ 8648.739735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8650.294115] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.18@tcp inode [0x200030971:0x5:0x0] object 0x280000401:11907 extent [0-4194303]: client csum a63c2b64, server csum eec88154 [ 8650.776739] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8653.857221] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8654.462298] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8658.207447] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 18:06:45 (1760134005) [ 8661.451891] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8663.801589] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8665.573907] Lustre: Failing over lustre-MDT0000 [ 8665.788430] Lustre: server umount lustre-MDT0000 complete [ 8666.891740] LustreError: 7451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760134014 with bad export cookie 16548930913753258863 [ 8666.894916] Lustre: Failing over lustre-OST0000 [ 8666.936962] Lustre: server umount lustre-OST0000 complete [ 8681.837583] LDISKFS-fs (dm-0): recovery complete [ 8681.839338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.193597] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe5a99cfc6ebe6a6b [ 8693.837308] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8697.468047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11764 to 0x2c0000401:11873) [ 8708.299971] LDISKFS-fs (dm-2): recovery complete [ 8708.301219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8710.338489] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8716.202315] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 18:07:43 (1760134063) [ 8721.072857] Lustre: Failing over lustre-OST0000 [ 8721.125162] Lustre: server umount lustre-OST0000 complete [ 8722.257909] Lustre: Failing over lustre-MDT0000 [ 8722.546631] Lustre: server umount lustre-MDT0000 complete [ 8735.726757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.142666] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8740.859741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11764 to 0x2c0000401:11905) [ 8747.026359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8749.324764] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8750.124726] Lustre: lustre-OST0000: Denying connection for new client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 8750.132725] Lustre: Skipped 1 previous similar message [ 8755.166710] Lustre: lustre-OST0000: Denying connection for new client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 8765.407421] Lustre: lustre-OST0000: Denying connection for new client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 8765.415985] Lustre: Skipped 1 previous similar message [ 8785.887702] Lustre: lustre-OST0000: Denying connection for new client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 8785.894753] Lustre: Skipped 3 previous similar messages [ 8818.000269] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8818.004269] Lustre: 255835:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client aa1af4ab-4869-4acf-ba69-8ec3324064d9@ [ 8818.011462] Lustre: 255835:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8818.016533] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8818.023656] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 8818.025869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11956 to 0x280000401:11977) [ 8818.029504] Lustre: Skipped 21 previous similar messages [ 8822.645798] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec [ 8827.034757] Lustre: DEBUG MARKER: free_before: 7646296 free_after: 7646296 [ 8829.038106] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 18:09:36 (1760134176) [ 8830.390542] Lustre: Failing over lustre-OST0000 [ 8830.413088] LustreError: 257072:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8830.416773] LustreError: 257072:0:(obd_class.h:479:obd_check_dev()) Skipped 163 previous similar messages [ 8830.433556] Lustre: server umount lustre-OST0000 complete [ 8831.972419] LustreError: 42153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.977383] LustreError: 42153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 251 previous similar messages [ 8833.503527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.506733] Lustre: Skipped 63 previous similar messages [ 8843.418294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8843.470665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.472778] Lustre: Skipped 22 previous similar messages [ 8843.475338] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 8843.476866] Lustre: Skipped 22 previous similar messages [ 8844.648408] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8844.650404] Lustre: Skipped 21 previous similar messages [ 8845.163644] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8849.002186] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 18:09:56 (1760134196) [ 8850.220230] Lustre: Failing over lustre-OST0000 [ 8850.260952] Lustre: server umount lustre-OST0000 complete [ 8863.554710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8865.642656] LustreError: 258815:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8865.645504] LustreError: 258815:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 77 previous similar messages [ 8865.659390] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8870.879262] Lustre: *** cfs_fail_loc=715, val=40*** [ 8881.119235] Lustre: 3657:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760134212/real 1760134212] req@ffffa030c556f800 x1845625383964800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1760134228 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 8881.136921] Lustre: 3657:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [ 8881.142876] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:52 [ 8881.149126] Lustre: Skipped 1 previous similar message [ 8882.078401] Lustre: lustre-OST0000: Client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp) reconnected, waiting for 3 clients in recovery for 0:51 [ 8887.263217] Lustre: *** cfs_fail_loc=715, val=40*** [ 8887.265099] Lustre: Skipped 2 previous similar messages [ 8888.287099] Lustre: *** cfs_fail_loc=715, val=40*** [ 8888.289057] Lustre: Skipped 1 previous similar message [ 8897.502673] Lustre: lustre-OST0000: Client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp) reconnected, waiting for 3 clients in recovery for 0:36 [ 8897.507163] Lustre: Skipped 2 previous similar messages [ 8903.647068] Lustre: *** cfs_fail_loc=715, val=40*** [ 8905.687139] LustreError: 258815:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8905.690569] LustreError: 258815:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 77 previous similar messages [ 8907.026855] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8907.562657] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8910.953537] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 18:10:58 (1760134258) [ 8912.232389] Lustre: Failing over lustre-MDT0000 [ 8912.494255] Lustre: server umount lustre-MDT0000 complete [ 8925.271525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8925.309624] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.312714] LustreError: Skipped 10 previous similar messages [ 8926.610165] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8930.793756] LustreError: 260257:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8934.367131] Lustre: *** cfs_fail_loc=715, val=80*** [ 8934.368249] Lustre: Skipped 2 previous similar messages [ 8943.582714] Lustre: lustre-MDT0000: Client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 8947.167419] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8949.727158] Lustre: *** cfs_fail_loc=715, val=80*** [ 8949.728397] Lustre: Skipped 1 previous similar message [ 8959.966560] Lustre: lustre-MDT0000: Client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 8963.551511] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8966.111188] Lustre: *** cfs_fail_loc=715, val=80*** [ 8966.112363] Lustre: Skipped 1 previous similar message [ 8976.350378] Lustre: lustre-MDT0000: Client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8978.911744] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8995.295619] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9001.439177] Lustre: *** cfs_fail_loc=715, val=80*** [ 9001.441213] Lustre: Skipped 4 previous similar messages [ 9008.094553] Lustre: lustre-MDT0000: Client b85dfc14-4cf9-464b-9d53-7985980be8c5 (at 192.168.201.18@tcp) reconnected, waiting for 2 clients in recovery for 0:22 [ 9008.097939] Lustre: Skipped 1 previous similar message [ 9010.879110] LustreError: 260257:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9010.884220] Lustre: 260257:0:(ldlm_lib.c:2929:target_recovery_thread()) too long recovery - read logs [ 9010.884302] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9010.887087] LustreError: dumping log to /tmp/lustre-log.1760134358.260257 [ 9010.888858] Lustre: Skipped 51 previous similar messages [ 9010.954579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11918 to 0x2c0000401:11937) [ 9010.954581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11991 to 0x280000401:12009) [ 9012.233713] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9012.802106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9016.404270] Lustre: DEBUG MARKER: == replay-single test complete, duration 8718 sec ======== 18:12:43 (1760134363) [ 9016.909753] Lustre: DEBUG MARKER: === replay-single: start cleanup 18:12:44 (1760134364) === [ 9020.255373] Lustre: DEBUG MARKER: === replay-single: finish cleanup 18:12:47 (1760134367) === [ 9020.928033] Lustre: Failing over lustre-MDT0000 [ 9021.146879] Lustre: server umount lustre-MDT0000 complete [ 9021.407502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9021.410184] LustreError: Skipped 13 previous similar messages [ 9036.948328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.472435] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9042.433545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11918 to 0x2c0000401:11969) [ 9042.433565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11991 to 0x280000401:12041) [ 9043.729292] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9044.272369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9052.640361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9052.642158] Lustre: Skipped 17 previous similar messages [ 9056.753315] Lustre: server umount lustre-MDT0000 complete [ 9059.683365] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760134407 with bad export cookie 16548930913753290685 [ 9059.687457] LustreError: 6511:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9059.825042] Lustre: server umount lustre-MDT0001 complete [ 9073.470785] Lustre: server umount lustre-OST0000 complete [ 9086.462397] Lustre: server umount lustre-OST0001 complete [ 9091.644436] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9092.666338] Key type lgssc unregistered [ 9092.800289] LNet: 264834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9092.802211] LNetError: 264834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9092.814342] LNet: Removed LNI 192.168.201.118@tcp [ 9093.142116] Key type .llcrypt unregistered [ 9093.143804] Key type ._llcrypt unregistered