[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515858575 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2528MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003339] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007649] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010150] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.021685] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026222] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] *** VALIDATE tmpfs *** [ 0.030480] *** VALIDATE proc *** [ 0.032228] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035080] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037024] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042089] debug: unmapping init [mem 0xffffffff98e59000-0xffffffff98e60fff] [ 0.044178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045756] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050020] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053368] rcu: Hierarchical SRCU implementation. [ 0.055650] smp: Bringing up secondary CPUs ... [ 0.056712] x86: Booting SMP configuration: [ 0.057032] .... node #0, CPUs: #1 #2 #3 [ 0.061013] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.139456] node 0 deferred pages initialised in 70ms [ 0.142186] devtmpfs: initialized [ 0.143266] x86/mm: Memory block size: 128MB [ 0.146253] gcov: version magic: 0x41383552 [ 0.148716] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154368] pinctrl core: initialized pinctrl subsystem [ 0.156339] [ 0.156851] ************************************************************* [ 0.160014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162076] ** ** [ 0.165013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.170010] ** This means that this kernel is built to expose internal ** [ 0.172012] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.176012] ** ** [ 0.178010] ** If you see this message and you are not debugging the ** [ 0.181014] ** kernel, report this immediately to your vendor! ** [ 0.183022] ** ** [ 0.186011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188010] ************************************************************* [ 0.191778] NET: Registered protocol family 16 [ 0.192478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201088] cpuidle: using governor menu [ 0.202786] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204593] PCI: Using configuration type 1 for base access [ 0.205099] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214020] cryptd: max_cpu_qlen set to 1000 [ 0.216235] ACPI: Added _OSI(Module Device) [ 0.217015] ACPI: Added _OSI(Processor Device) [ 0.218014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223045] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227450] ACPI: Interpreter enabled [ 0.228041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229000] ACPI: Using IOAPIC for interrupt routing [ 0.229000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240824] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245000] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249496] acpiphp: Slot [2] registered [ 0.250148] acpiphp: Slot [5] registered [ 0.252147] acpiphp: Slot [6] registered [ 0.254090] acpiphp: Slot [7] registered [ 0.255140] acpiphp: Slot [8] registered [ 0.256137] acpiphp: Slot [9] registered [ 0.258075] acpiphp: Slot [10] registered [ 0.259109] acpiphp: Slot [3] registered [ 0.261144] acpiphp: Slot [4] registered [ 0.263109] acpiphp: Slot [11] registered [ 0.264177] acpiphp: Slot [12] registered [ 0.266104] acpiphp: Slot [13] registered [ 0.268109] acpiphp: Slot [14] registered [ 0.269100] acpiphp: Slot [15] registered [ 0.271109] acpiphp: Slot [16] registered [ 0.272128] acpiphp: Slot [17] registered [ 0.273145] acpiphp: Slot [18] registered [ 0.275155] acpiphp: Slot [19] registered [ 0.276199] acpiphp: Slot [20] registered [ 0.278097] acpiphp: Slot [21] registered [ 0.279093] acpiphp: Slot [22] registered [ 0.281100] acpiphp: Slot [23] registered [ 0.282104] acpiphp: Slot [24] registered [ 0.284108] acpiphp: Slot [25] registered [ 0.285088] acpiphp: Slot [26] registered [ 0.286122] acpiphp: Slot [27] registered [ 0.288092] acpiphp: Slot [28] registered [ 0.289097] acpiphp: Slot [29] registered [ 0.290146] acpiphp: Slot [30] registered [ 0.292101] acpiphp: Slot [31] registered [ 0.293170] PCI host bridge to bus 0000:00 [ 0.295020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309620] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311970] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323448] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.328041] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.331318] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334198] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.338048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.340946] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.346012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.353964] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.359016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.402724] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.411020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.418019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.433021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.442957] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.453018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.460016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.483018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.492747] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.498019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.505019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.520024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.530188] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.536013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.541017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.554017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.564496] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.579018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.591023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.612018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.622463] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.625402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.627474] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630517] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.633238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.638034] iommu: Default domain type: Passthrough [ 0.640416] SCSI subsystem initialized [ 0.642212] ACPI: bus type USB registered [ 0.643119] usbcore: registered new interface driver usbfs [ 0.645080] usbcore: registered new interface driver hub [ 0.647073] usbcore: registered new device driver usb [ 0.648151] pps_core: LinuxPPS API ver. 1 registered [ 0.650013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.653078] PTP clock support registered [ 0.655172] EDAC MC: Ver: 3.0.0 [ 0.658127] PCI: Using ACPI for IRQ routing [ 0.661224] NetLabel: Initializing [ 0.662011] NetLabel: domain hash size = 128 [ 0.664012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.665084] NetLabel: unlabeled traffic allowed by default [ 0.667131] vgaarb: loaded [ 0.668400] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.671019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.678006] clocksource: Switched to clocksource kvm-clock [ 0.793843] VFS: Disk quotas dquot_6.6.0 [ 0.795601] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.798022] *** VALIDATE ramfs *** [ 0.799463] *** VALIDATE hugetlbfs *** [ 0.801038] pnp: PnP ACPI init [ 0.803340] pnp: PnP ACPI: found 6 devices [ 0.824923] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.828387] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.830653] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.832995] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.835491] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.838043] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.841015] NET: Registered protocol family 2 [ 0.843534] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.848291] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.851758] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.856637] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.859610] TCP: Hash tables configured (established 65536 bind 65536) [ 0.861979] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.864838] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.867075] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869949] NET: Registered protocol family 1 [ 0.872396] RPC: Registered named UNIX socket transport module. [ 0.873813] RPC: Registered udp transport module. [ 0.874911] RPC: Registered tcp transport module. [ 0.875893] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.877529] NET: Registered protocol family 44 [ 0.878986] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.882555] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885481] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888786] PCI: CLS 0 bytes, default 64 [ 0.892078] Unpacking initramfs... [ 2.268337] debug: unmapping init [mem 0xffff9dbdfcc54000-0xffff9dbdfffbffff] [ 2.272483] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.274847] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.278016] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.802847] Initialise system trusted keyrings [ 2.804514] Key type blacklist registered [ 2.805989] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.815126] zbud: loaded [ 2.817659] *** VALIDATE nfs *** [ 2.818586] *** VALIDATE nfs4 *** [ 2.819794] pstore: using deflate compression [ 2.822517] Platform Keyring initialized [ 2.902503] NET: Registered protocol family 38 [ 2.903913] Key type asymmetric registered [ 2.905083] Asymmetric key parser 'x509' registered [ 2.906456] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.909176] io scheduler mq-deadline registered [ 2.911060] io scheduler kyber registered [ 2.912712] io scheduler bfq registered [ 2.915093] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.917482] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.919920] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.922810] ACPI: Power Button [PWRF] [ 2.927361] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.932575] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.939975] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.944108] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.953071] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.979241] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.005789] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.009126] Non-volatile memory driver v1.3 [ 3.010192] Linux agpgart interface v0.103 [ 3.035854] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.037776] vda: detected capacity change from 0 to 74326016 [ 3.051185] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.053080] vdb: detected capacity change from 0 to 1073741824 [ 3.064445] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.067116] vdc: detected capacity change from 0 to 2621440000 [ 3.081148] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.083087] vdd: detected capacity change from 0 to 2621440000 [ 3.093055] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.094797] vde: detected capacity change from 0 to 4294967296 [ 3.105701] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.107731] vdf: detected capacity change from 0 to 4294967296 [ 3.112159] libphy: Fixed MDIO Bus: probed [ 3.117704] usbcore: registered new interface driver usbserial_generic [ 3.120721] usbserial: USB Serial support registered for generic [ 3.122827] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.126347] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.127540] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.129360] mousedev: PS/2 mouse device common for all mice [ 3.131423] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.133174] rtc_cmos 00:05: RTC can wake from S4 [ 3.136550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.142719] rtc_cmos 00:05: registered as rtc0 [ 3.144219] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.144744] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.148940] intel_pstate: CPU model not supported [ 3.150663] hid: raw HID events driver (C) Jiri Kosina [ 3.152371] usbcore: registered new interface driver usbhid [ 3.154537] usbhid: USB HID core driver [ 3.156153] drop_monitor: Initializing network drop monitor service [ 3.157696] Initializing XFRM netlink socket [ 3.158923] NET: Registered protocol family 10 [ 3.161188] Segment Routing with IPv6 [ 3.162121] NET: Registered protocol family 17 [ 3.163415] mpls_gso: MPLS GSO support [ 3.167387] RAS: Correctable Errors collector initialized. [ 3.168641] AVX version of gcm_enc/dec engaged. [ 3.170126] AES CTR mode by8 optimization enabled [ 3.238869] sched_clock: Marking stable (3238853084, 0)->(4163078692, -924225608) [ 3.241725] registered taskstats version 1 [ 3.243072] Loading compiled-in X.509 certificates [ 3.244559] zswap: loaded using pool lzo/zbud [ 3.261809] Key type big_key registered [ 3.270439] Key type encrypted registered [ 3.271634] ima: No TPM chip found, activating TPM-bypass! [ 3.272841] ima: Allocated hash algorithm: sha1 [ 3.274169] ima: No architecture policies found [ 3.275288] evm: Initialising EVM extended attributes: [ 3.276393] evm: security.selinux [ 3.276963] evm: security.ima [ 3.277673] evm: security.capability [ 3.278654] evm: HMAC attrs: 0x1 [ 3.280410] rtc_cmos 00:05: setting system clock to 2026-07-04 04:08:16 UTC (1783138096) [ 3.285259] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 3.287192] debug: unmapping init [mem 0xffffffff98b82000-0xffffffff98e58fff] [ 3.299960] Write protecting the kernel read-only data: 28672k [ 3.303263] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.305598] debug: unmapping init [mem 0xffffffff97b14000-0xffffffff97bfffff] [ 3.332343] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.337386] systemd[1]: Detected virtualization kvm. [ 3.338629] systemd[1]: Detected architecture x86-64. [ 3.339780] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.364673] systemd[1]: No hostname configured. [ 3.365713] systemd[1]: Set hostname to . [ 3.367107] random: systemd: uninitialized urandom read (16 bytes read) [ 3.368594] systemd[1]: Initializing machine ID from random generator. [ 3.425325] random: ln: uninitialized urandom read (6 bytes read) [ 3.504979] random: systemd: uninitialized urandom read (16 bytes read) [ 3.507811] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.513575] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.517036] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.239044] device-mapper: uevent: version 1.0.3 [ 4.241665] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.065383] virtio_net virtio0 ens2: renamed from eth0 [ 5.124238] scsi host0: ata_piix [ 5.129572] random: fast init done [ 5.186226] scsi host1: ata_piix [ 5.189743] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.192449] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.693836] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.783549] random: crng init done [ 9.785024] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.371668] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.751436] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.004454] SELinux: Disabled at runtime. [ 12.066714] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.077255] systemd[1]: Detected virtualization kvm. [ 12.079527] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.684689] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.689775] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.696662] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.712736] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.718148] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.730574] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.736526] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ 12.882859] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.462869] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.899309] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.981414] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.083524] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.109063] EDAC sbridge: Ver: 1.1.2 [ 15.817455] Key type dns_resolver registered [ 16.171224] NFS: Registering the id_resolver key type [ 16.173619] Key type id_resolver registered [ 16.175665] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-server login: [ 63.981513] libcfs: loading out-of-tree module taints kernel. [ 64.014884] Key type ._llcrypt registered [ 64.016934] Key type .llcrypt registered [ 64.104056] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 86.152905] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 88.077108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.121613] alg: No test for adler32 (adler32-zlib) [ 90.007880] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 91.073843] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 92.903153] Key type lgssc registered [ 95.829626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 103.202088] hrtimer: interrupt took 10041054 ns [ 116.046792] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.794728] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 175.146871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.180159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.401271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.441508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.555966] Lustre: lustre-MDT0000: new disk, initializing [ 176.670706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.704257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.175964] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.818266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.931537] Lustre: 6510:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.960096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.965132] Lustre: Skipped 1 previous similar message [ 195.092702] Lustre: lustre-MDT0001: new disk, initializing [ 195.187143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.242444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.251746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.310990] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.110316] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 218.898282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.216817] Lustre: lustre-OST0000: new disk, initializing [ 219.221243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.230989] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.343469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.277399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.292109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.396977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 227.404698] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.978295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.202541] Lustre: lustre-OST0001: new disk, initializing [ 245.205742] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.210414] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.321648] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 253.251505] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.496913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.510495] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.577488] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.685626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.055602] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 288.602487] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing check_logdir /tmp/testlogs/ [ 295.631964] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing yml_node [ 300.902745] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 304.055292] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 307.064617] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 309.211862] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat Jul 4 00:13:20 EDT 2026 [ 328.216507] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 330.439944] Lustre: DEBUG MARKER: === replay-single: start setup 00:13:41 (1783138421) === [ 336.769071] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing check_config_client /mnt/lustre [ 357.766804] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 361.950334] Lustre: 13336:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 365.856274] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 371.411580] Lustre: DEBUG MARKER: === replay-single: finish setup 00:14:23 (1783138463) === [ 373.457844] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 00:14:24 (1783138464) [ 383.164613] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 386.149331] Lustre: Failing over lustre-MDT0000 [ 386.532636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.732717] Lustre: server umount lustre-MDT0000 complete [ 391.649989] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.678527] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 394.915339] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.769413] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.788809] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 400.030581] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.953315] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.991732] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 407.014773] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783138484/real 1783138484] req@ffff9dbd43ebc000 x1869755904556160/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783138500 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 407.047242] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 411.141817] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 411.144419] LDISKFS-fs (dm-0): recovery complete [ 411.153667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.406819] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.427897] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 417.265351] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe7523a300 x1869755904564224/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 417.552327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.399755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 422.207734] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 422.925726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 422.970821] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 434.457060] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 436.728317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 447.122424] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 00:15:38 (1783138538) [ 449.670990] Lustre: Failing over lustre-OST0000 [ 449.748302] Lustre: server umount lustre-OST0000 complete [ 450.235605] LustreError: 8432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.260943] LustreError: 8432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 451.041405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.058937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.076696] Lustre: Skipped 3 previous similar messages [ 453.605151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.934130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 470.249509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.689497] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 472.394301] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 472.400469] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 472.416226] Lustre: Skipped 3 previous similar messages [ 476.647818] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 486.662549] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 488.331282] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 499.984708] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 00:16:31 (1783138591) [ 509.159082] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 511.818233] Lustre: Failing over lustre-MDT0000 [ 512.292941] Lustre: server umount lustre-MDT0000 complete [ 515.553750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 515.568706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.612061] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.641398] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 531.872766] Lustre: 3642:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783138609/real 1783138609] req@ffff9dbe71b63800 x1869755904626688/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783138625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 531.926120] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.836622] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 535.839196] LDISKFS-fs (dm-0): recovery complete [ 535.844353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.601231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.461727] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 547.827490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 547.833363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 547.847086] Lustre: Skipped 1 previous similar message [ 550.043548] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 555.174650] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 560.297959] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 565.412563] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 570.526067] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 580.769158] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:36 [ 580.806289] Lustre: Skipped 1 previous similar message [ 601.256495] Lustre: lustre-MDT0000: Denying connection for new client 750f758c-98c6-42f0-aeec-184f68d9f244 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 601.273838] Lustre: Skipped 3 previous similar messages [ 617.500206] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 617.503455] Lustre: 18243:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c26ac06a-3f1c-49f9-821d-fc478a16795e@ [ 617.547560] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 617.572381] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 617.578378] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 617.614671] Lustre: Skipped 2 previous similar messages [ 617.647274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 617.650625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 632.114532] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 00:18:43 (1783138723) [ 640.084596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 642.532038] Lustre: Failing over lustre-MDT0000 [ 643.044233] Lustre: server umount lustre-MDT0000 complete [ 643.553961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.558902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.565134] Lustre: Skipped 3 previous similar messages [ 643.572566] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 643.586285] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 661.473129] Lustre: 3642:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783138738/real 1783138738] req@ffff9dbd4301ad80 x1869755904687360/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783138754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 661.491655] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.484608] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 669.494048] LDISKFS-fs (dm-0): recovery complete [ 669.536952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.817672] LustreError: 19937:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 670.833578] LustreError: 19937:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9dbe7b44fb80 x1869755904693120/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783138764 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 670.873109] LustreError: 19937:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 671.715687] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b077fb66 [ 671.741164] Lustre: MGC192.168.202.153@tcp: Connection restored to 0@lo (at 0@lo) [ 672.115721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.195846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 677.188092] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 677.352023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 679.886543] Lustre: lustre-MDT0000: Denying connection for new client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 679.911730] Lustre: Skipped 3 previous similar messages [ 746.655339] Lustre: lustre-MDT0000: Denying connection for new client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 746.673646] Lustre: Skipped 12 previous similar messages [ 747.500359] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 747.505062] Lustre: 19972:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 750f758c-98c6-42f0-aeec-184f68d9f244@ [ 747.521377] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 747.545166] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 747.545270] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 747.556633] Lustre: Skipped 3 previous similar messages [ 747.593539] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 747.596769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 760.243595] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 00:20:52 (1783138852) [ 766.757665] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 768.437882] Lustre: Failing over lustre-MDT0000 [ 768.608581] Lustre: server umount lustre-MDT0000 complete [ 769.506266] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.527829] Lustre: Skipped 4 previous similar messages [ 772.289642] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.308214] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 32 previous similar messages [ 784.864332] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783138862/real 1783138862] req@ffff9dbe44753800 x1869755904744320/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783138878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 784.893309] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.140608] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 790.142906] LDISKFS-fs (dm-0): recovery complete [ 790.148239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.365164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.416343] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 797.859145] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 799.988164] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 800.751503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 800.824309] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 800.895031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 800.908554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 809.601700] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.702328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 822.208079] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 00:21:53 (1783138913) [ 830.280823] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 832.396456] Lustre: Failing over lustre-MDT0000 [ 832.589217] Lustre: server umount lustre-MDT0000 complete [ 836.590341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.604965] Lustre: Skipped 5 previous similar messages [ 851.935242] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783138929/real 1783138929] req@ffff9dbd4209c700 x1869755904779648/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783138945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 851.948258] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.432325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 854.440327] LDISKFS-fs (dm-0): recovery complete [ 854.449439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.178042] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b0780545 [ 862.190348] Lustre: MGC192.168.202.153@tcp: Connection restored to 0@lo (at 0@lo) [ 862.196380] Lustre: Skipped 3 previous similar messages [ 862.534033] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 863.622365] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 866.685066] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 867.965058] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 868.004154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 868.004525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 875.449738] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.153839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 885.126930] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 00:22:57 (1783138977) [ 892.144418] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 893.782473] Lustre: Failing over lustre-MDT0000 [ 893.925114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 893.930469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.972842] Lustre: server umount lustre-MDT0000 complete [ 913.528581] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 913.534550] LDISKFS-fs (dm-0): recovery complete [ 913.546727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.614229] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.801642] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 914.609775] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 917.759720] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 919.127050] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 919.174717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 919.176182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 926.876394] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 928.300978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 936.761516] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 00:23:48 (1783139028) [ 944.475682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 946.519110] Lustre: Failing over lustre-MDT0000 [ 946.751977] Lustre: server umount lustre-MDT0000 complete [ 965.092124] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139042/real 1783139042] req@ffff9dbe75d00e00 x1869755904845184/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 965.138546] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 972.010628] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 972.015950] LDISKFS-fs (dm-0): recovery complete [ 972.039608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.329821] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbd44d14e00 x1869755904853760/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 975.664328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.670972] Lustre: Skipped 2 previous similar messages [ 975.727587] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 977.586368] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 980.659763] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 980.969154] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 980.986255] Lustre: Skipped 8 previous similar messages [ 981.129786] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 981.185766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 981.186016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 989.386602] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 990.626435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 998.847583] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 00:24:50 (1783139090) [ 1005.687168] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1007.411911] Lustre: Failing over lustre-MDT0000 [ 1007.643338] Lustre: server umount lustre-MDT0000 complete [ 1011.680937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.698971] Lustre: Skipped 9 previous similar messages [ 1027.535343] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139104/real 1783139104] req@ffff9dbe44195f80 x1869755904880128/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1027.552138] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1029.628774] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1029.631410] LDISKFS-fs (dm-0): recovery complete [ 1029.637680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.793903] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.807749] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 124 previous similar messages [ 1037.791479] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe6f3bce00 x1869755904887680/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1038.029134] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1043.226338] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1043.609077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1043.613427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1053.733245] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1055.765874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1064.732413] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 00:25:56 (1783139156) [ 1065.893148] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1065.895897] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1065.900448] LustreError: 6518:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe6feddf80 x1869755881507968/t38654705666(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:555/0 lens 392/456 e 0 to 0 dl 1783139170 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1074.173496] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1076.015337] Lustre: Failing over lustre-MDT0000 [ 1076.484954] Lustre: server umount lustre-MDT0000 complete [ 1095.647236] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139172/real 1783139172] req@ffff9dbe71b62d80 x1869755904919040/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1095.668647] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.727736] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1100.730964] LDISKFS-fs (dm-0): recovery complete [ 1100.745948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.870317] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b07819ce [ 1105.207486] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1106.606219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1106.621958] Lustre: Skipped 1 previous similar message [ 1109.570462] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1110.519793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1110.536980] Lustre: Skipped 8 previous similar messages [ 1110.589451] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1110.604888] Lustre: Skipped 1 previous similar message [ 1110.613249] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe7f540000 x1869755881507968/t38654705666(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:599/0 lens 392/456 e 0 to 0 dl 1783139214 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1110.663405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1110.676210] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1119.467757] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1121.380363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1131.094137] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 00:27:02 (1783139222) [ 1139.498151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1142.288565] Lustre: Failing over lustre-MDT0000 [ 1142.444562] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 1142.771896] Lustre: server umount lustre-MDT0000 complete [ 1146.336100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1146.344382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.359792] Lustre: Skipped 8 previous similar messages [ 1162.722482] Lustre: 3641:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139239/real 1783139239] req@ffff9dbe6f37ca80 x1869755904957824/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1166.084516] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1166.086820] LDISKFS-fs (dm-0): recovery complete [ 1166.092675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.369110] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1177.779741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1177.796343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1178.170262] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1188.950516] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1191.016787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1200.623979] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 00:28:12 (1783139292) [ 1208.477180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1209.406721] Lustre: *** cfs_fail_loc=114, val=0*** [ 1212.139361] Lustre: Failing over lustre-MDT0000 [ 1212.436198] Lustre: server umount lustre-MDT0000 complete [ 1229.791367] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.798286] LustreError: Skipped 1 previous similar message [ 1234.618887] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1234.622502] LDISKFS-fs (dm-0): recovery complete [ 1234.634942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.328313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.338489] Lustre: Skipped 3 previous similar messages [ 1240.387667] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1240.742883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1240.753246] Lustre: Skipped 1 previous similar message [ 1244.943810] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1245.774112] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1245.782926] Lustre: Skipped 1 previous similar message [ 1245.839143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1245.854345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1255.924462] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.982679] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1266.706921] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 00:29:18 (1783139358) [ 1274.319794] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1275.379782] Lustre: *** cfs_fail_loc=128, val=0*** [ 1277.930415] Lustre: Failing over lustre-MDT0000 [ 1278.234748] Lustre: server umount lustre-MDT0000 complete [ 1281.515810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1296.799208] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139374/real 1783139374] req@ffff9dbe44286680 x1869755905033472/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1296.826281] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1300.146302] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1300.149947] LDISKFS-fs (dm-0): recovery complete [ 1300.156687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.296994] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (not set up) [ 1312.519800] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1312.846952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1312.856148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1322.758424] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.346403] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1333.353430] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 00:30:24 (1783139424) [ 1341.244702] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1343.178920] Lustre: Failing over lustre-MDT0000 [ 1343.599462] Lustre: server umount lustre-MDT0000 complete [ 1367.297856] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1367.303273] LDISKFS-fs (dm-0): recovery complete [ 1367.328834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.176769] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe4492f480 x1869755905076096/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1374.565415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1374.579048] Lustre: Skipped 1 previous similar message [ 1379.080587] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1379.855673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1379.892800] Lustre: Skipped 15 previous similar messages [ 1380.281397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1380.287284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1390.278769] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1392.409111] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1403.321164] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 00:31:34 (1783139494) [ 1413.697599] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1416.418681] Lustre: Failing over lustre-MDT0000 [ 1416.941169] Lustre: server umount lustre-MDT0000 complete [ 1420.770495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.789963] Lustre: Skipped 14 previous similar messages [ 1439.969742] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1439.976334] LDISKFS-fs (dm-0): recovery complete [ 1439.991929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.304071] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1452.191257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1452.191723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1460.983996] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1462.473335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1471.063763] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 00:32:42 (1783139562) [ 1479.150915] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1486.771340] Lustre: Failing over lustre-MDT0000 [ 1487.199930] Lustre: server umount lustre-MDT0000 complete [ 1504.227778] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1504.243928] LustreError: Skipped 3 previous similar messages [ 1513.077976] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1513.080850] LDISKFS-fs (dm-0): recovery complete [ 1513.095552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.272282] LustreError: 43255:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1514.279182] LustreError: 43255:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9dbe6f024700 x1869755905153280/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783139607 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1514.294111] LustreError: 43255:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1514.470137] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b0787044 [ 1515.679426] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1515.691025] Lustre: Skipped 3 previous similar messages [ 1519.933979] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1523.507781] Lustre: lustre-MDT0000: Recovery over after 0:08, of 2 clients 2 recovered and 0 were evicted. [ 1523.521485] Lustre: Skipped 3 previous similar messages [ 1523.568491] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1523.576306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1529.650049] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1531.539625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1559.253972] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 00:34:10 (1783139650) [ 1568.096378] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1570.587358] Lustre: Failing over lustre-MDT0000 [ 1570.941185] Lustre: server umount lustre-MDT0000 complete [ 1571.312826] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.350538] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 237 previous similar messages [ 1587.691125] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139664/real 1783139664] req@ffff9dbe5ef24e00 x1869755905248768/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783139680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1587.722639] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1594.390952] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1594.393043] LDISKFS-fs (dm-0): recovery complete [ 1594.407734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.906796] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1602.676945] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1602.691277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1610.968507] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.503748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1626.010319] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 00:35:16 (1783139716) [ 1636.125369] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1638.725263] Lustre: Failing over lustre-MDT0000 [ 1638.864075] Lustre: server umount lustre-MDT0000 complete [ 1643.499750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.180729] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1662.184210] LDISKFS-fs (dm-0): recovery complete [ 1662.193402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.388406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1669.399744] Lustre: Skipped 3 previous similar messages [ 1673.806713] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1674.857538] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1674.858784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1681.881027] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.560944] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1691.229958] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 00:36:23 (1783139783) [ 1697.947617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1700.175308] Lustre: Failing over lustre-MDT0000 [ 1700.321640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1700.331703] Lustre: Skipped 3 previous similar messages [ 1700.610388] Lustre: server umount lustre-MDT0000 complete [ 1722.731071] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1722.734293] LDISKFS-fs (dm-0): recovery complete [ 1722.744112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.555672] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe44750a80 x1869755905332864/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1736.103242] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1737.286565] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1737.289560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1745.282199] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1746.905521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1755.929904] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 00:37:27 (1783139847) [ 1764.803730] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1767.508642] Lustre: Failing over lustre-MDT0000 [ 1767.903865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1769.874940] Lustre: server umount lustre-MDT0000 complete [ 1793.483970] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1793.486524] LDISKFS-fs (dm-0): recovery complete [ 1793.494256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.657942] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b078e6d4 [ 1800.093493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.096837] Lustre: Skipped 7 previous similar messages [ 1804.765993] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1805.555468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1805.557077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1817.583469] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1820.107866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1829.947496] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 00:38:41 (1783139921) [ 1838.631663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1840.683057] Lustre: Failing over lustre-MDT0000 [ 1842.942671] Lustre: server umount lustre-MDT0000 complete [ 1865.421352] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1865.423129] LDISKFS-fs (dm-0): recovery complete [ 1865.434177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.231631] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe5ef25180 x1869755905405696/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1873.039407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1873.043351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1873.344932] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1885.683331] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.896784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1897.462367] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 00:39:49 (1783139989) [ 1906.020689] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1908.291909] Lustre: Failing over lustre-MDT0000 [ 1908.731724] Lustre: server umount lustre-MDT0000 complete [ 1932.089725] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1932.091750] LDISKFS-fs (dm-0): recovery complete [ 1932.109715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.448456] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe75d00700 x1869755905444224/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1940.468220] Lustre: MGS: Client 472c019f-e14b-4bee-bf72-0973e6fbd974 (at 0@lo) reconnecting [ 1940.481698] Lustre: MGC192.168.202.153@tcp: Connection restored to 0@lo (at 0@lo) [ 1940.490601] Lustre: Skipped 33 previous similar messages [ 1945.287604] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1946.226785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1946.234433] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1955.273362] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1957.019649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1965.851570] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 00:40:57 (1783140057) [ 1974.732746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1977.074656] Lustre: Failing over lustre-MDT0000 [ 1977.276826] Lustre: server umount lustre-MDT0000 complete [ 1981.924320] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.925546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1981.941849] Lustre: Skipped 30 previous similar messages [ 2000.215438] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2000.217630] LDISKFS-fs (dm-0): recovery complete [ 2000.223607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.150169] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2014.443121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 2014.445802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 2024.577638] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2026.963686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2039.102444] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 00:42:10 (1783140130) [ 2051.456259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2054.335054] Lustre: Failing over lustre-MDT0000 [ 2055.489916] Lustre: server umount lustre-MDT0000 complete [ 2071.523106] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2071.529873] LustreError: Skipped 8 previous similar messages [ 2079.408978] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2079.421756] LDISKFS-fs (dm-0): recovery complete [ 2079.448473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.631232] LustreError: 58806:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2081.640681] LustreError: 58806:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9dbe6f2b0700 x1869755905518080/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783140174 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2081.661519] LustreError: 58806:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2081.765739] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b078fd2b [ 2082.805427] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2082.820259] Lustre: Skipped 7 previous similar messages [ 2086.676464] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2087.435611] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2087.444080] Lustre: Skipped 7 previous similar messages [ 2087.499286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2087.500279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2097.273372] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2099.204619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2110.557299] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 00:43:22 (1783140202) [ 2118.140221] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2120.313613] Lustre: Failing over lustre-MDT0000 [ 2120.558454] Lustre: server umount lustre-MDT0000 complete [ 2123.233276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.088865] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783140216/real 1783140216] req@ffff9dbe80ef5180 x1869755905550080/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783140232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2139.126487] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2143.786422] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2143.789819] LDISKFS-fs (dm-0): recovery complete [ 2143.796764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.842323] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2155.073631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 2155.073708] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 2163.672632] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2165.963818] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2175.639759] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 00:44:27 (1783140267) [ 2183.188856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2185.678968] Lustre: Failing over lustre-MDT0000 [ 2185.710137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.713884] Lustre: Skipped 1 previous similar message [ 2186.010979] Lustre: server umount lustre-MDT0000 complete [ 2186.439343] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.457989] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 316 previous similar messages [ 2209.226316] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2209.234076] LDISKFS-fs (dm-0): recovery complete [ 2209.244259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.335340] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b07907dc [ 2212.724712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2212.741056] Lustre: Skipped 7 previous similar messages [ 2217.228363] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2218.064931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 2218.065464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 2226.601124] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2228.201501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2238.215196] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 00:45:29 (1783140329) [ 2248.087955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2250.498753] Lustre: Failing over lustre-MDT0000 [ 2250.957129] Lustre: server umount lustre-MDT0000 complete [ 2275.606456] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2275.609371] LDISKFS-fs (dm-0): recovery complete [ 2275.619446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.914613] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b0790d54 [ 2285.752034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2285.753317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2285.989873] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2296.487241] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2298.337940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2307.237882] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 00:46:39 (1783140399) [ 2314.821817] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2317.162786] Lustre: Failing over lustre-MDT0000 [ 2317.492053] Lustre: server umount lustre-MDT0000 complete [ 2339.405613] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2339.408116] LDISKFS-fs (dm-0): recovery complete [ 2339.420968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.978847] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b07912da [ 2352.033718] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2352.732742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 2352.735483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 2360.474961] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2361.970674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2369.429376] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 00:47:41 (1783140461) [ 2377.265338] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2379.621922] Lustre: Failing over lustre-MDT0000 [ 2379.794626] Lustre: server umount lustre-MDT0000 complete [ 2383.329832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2383.337635] LustreError: Skipped 1 previous similar message [ 2401.976947] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2401.978679] LDISKFS-fs (dm-0): recovery complete [ 2401.988792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.466375] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe4493e300 x1869755905707904/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2410.704922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2410.708566] Lustre: Skipped 8 previous similar messages [ 2415.613861] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2416.253243] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:993) [ 2416.263560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2425.396827] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2426.996306] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2436.256491] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 00:48:47 (1783140527) [ 2443.827198] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2445.703196] Lustre: Failing over lustre-MDT0000 [ 2447.963498] Lustre: server umount lustre-MDT0000 complete [ 2468.329803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2468.333347] LDISKFS-fs (dm-0): recovery complete [ 2468.341075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.752245] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2479.213521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2479.222957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2486.639146] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2488.021361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2496.138184] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 00:49:48 (1783140588) [ 2502.919247] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2504.693973] Lustre: Failing over lustre-MDT0000 [ 2504.889206] Lustre: server umount lustre-MDT0000 complete [ 2527.572211] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2527.575795] LDISKFS-fs (dm-0): recovery complete [ 2527.593156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.594800] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2542.063523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2542.069362] Lustre: Skipped 40 previous similar messages [ 2542.207202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2542.209736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2548.188779] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2549.596574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2557.545549] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:50:49 (1783140649) [ 2563.907624] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2565.839835] Lustre: Failing over lustre-MDT0000 [ 2566.031689] Lustre: server umount lustre-MDT0000 complete [ 2588.303442] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2588.305708] LDISKFS-fs (dm-0): recovery complete [ 2588.313541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.249108] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe444f5500 x1869755905810304/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2597.649352] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2598.992075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 2598.993225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 2606.591266] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2608.346341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2616.421353] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 00:51:48 (1783140708) [ 2621.130255] Lustre: 75337:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ec529461-6aae-4389-8044-443dfe589617 at adminstrative request [ 2627.501323] Lustre: Failing over lustre-MDT0000 [ 2627.516206] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 2627.689647] Lustre: server umount lustre-MDT0000 complete [ 2629.600335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.622922] Lustre: Skipped 39 previous similar messages [ 2645.486422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.499604] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2651.167416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2651.167634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 2651.310932] LustreError: 42744:0:(ofd_io.c:785:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x280000401:1090 [ 2651.316868] LustreError: 42744:0:(ofd_io.c:785:ofd_preprw_write()) Skipped 1 previous similar message [ 2658.745283] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2660.294717] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2665.913498] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2674.445901] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2680.472540] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 00:52:52 (1783140772) [ 2681.422542] Lustre: 77362:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ec529461-6aae-4389-8044-443dfe589617 at adminstrative request [ 2690.945310] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 00:53:02 (1783140782) [ 2697.811332] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2699.424714] Lustre: Failing over lustre-MDT0000 [ 2699.668912] Lustre: server umount lustre-MDT0000 complete [ 2718.687301] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.696427] LustreError: Skipped 9 previous similar messages [ 2721.744492] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2721.747814] LDISKFS-fs (dm-0): recovery complete [ 2721.753904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.657781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2730.670694] Lustre: Skipped 9 previous similar messages [ 2732.588058] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2733.671595] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2733.681891] Lustre: Skipped 9 previous similar messages [ 2733.721344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2733.725080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2742.801379] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2744.602956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2754.038964] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:54:05 (1783140845) [ 2761.994884] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2764.150422] Lustre: Failing over lustre-MDT0000 [ 2764.265563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.275361] Lustre: Skipped 3 previous similar messages [ 2764.674549] Lustre: server umount lustre-MDT0000 complete [ 2785.760655] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783140862/real 1783140862] req@ffff9dbe4452d180 x1869755905915392/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783140878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2785.791671] Lustre: 3643:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2786.095650] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2786.098273] LDISKFS-fs (dm-0): recovery complete [ 2786.105080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.008255] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.029920] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 312 previous similar messages [ 2796.008187] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b0793abf [ 2800.533829] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2801.776216] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2801.776415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2810.211593] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2811.771687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2820.197268] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 00:55:11 (1783140911) [ 2827.993146] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2829.788515] Lustre: Failing over lustre-MDT0000 [ 2829.996301] Lustre: server umount lustre-MDT0000 complete [ 2851.216126] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2851.219910] LDISKFS-fs (dm-0): recovery complete [ 2851.233566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.952105] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe44886300 x1869755905959296/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2858.352144] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2858.365653] Lustre: Skipped 9 previous similar messages [ 2862.832320] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2863.719836] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2863.720394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2871.851024] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2873.526806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2881.955910] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 00:56:13 (1783140973) [ 2889.848462] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2892.189617] Lustre: Failing over lustre-MDT0000 [ 2892.462855] Lustre: server umount lustre-MDT0000 complete [ 2913.645776] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2913.648457] LDISKFS-fs (dm-0): recovery complete [ 2913.653993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.395310] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2925.605197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2925.616361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2933.473503] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2935.009169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2942.927526] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:57:14 (1783141034) [ 2950.085910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2952.296743] Lustre: Failing over lustre-MDT0000 [ 2952.588429] Lustre: server umount lustre-MDT0000 complete [ 2956.256672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2956.264456] LustreError: Skipped 5 previous similar messages [ 2972.569894] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2972.572388] LDISKFS-fs (dm-0): recovery complete [ 2972.580689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.457949] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2987.572627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2987.572627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2993.919097] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2995.175754] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3002.282848] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 00:58:14 (1783141094) [ 3009.220295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3011.040661] Lustre: Failing over lustre-MDT0000 [ 3011.045337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.053032] Lustre: Skipped 2 previous similar messages [ 3011.300750] Lustre: server umount lustre-MDT0000 complete [ 3031.569334] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3031.571519] LDISKFS-fs (dm-0): recovery complete [ 3031.576749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.909664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.917577] Lustre: Skipped 9 previous similar messages [ 3035.663047] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3037.253449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 3037.253699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 3043.446458] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3045.115730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3052.438496] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:59:04 (1783141144) [ 3059.146976] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3061.053341] Lustre: Failing over lustre-MDT0000 [ 3061.284544] Lustre: server umount lustre-MDT0000 complete [ 3081.094289] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3081.099985] LDISKFS-fs (dm-0): recovery complete [ 3081.109263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.329188] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3094.129892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3094.129970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3099.885072] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3101.301221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3108.551660] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 01:00:00 (1783141200) [ 3115.517914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3117.406090] Lustre: Failing over lustre-MDT0000 [ 3117.737156] Lustre: server umount lustre-MDT0000 complete [ 3138.385199] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3138.389024] LDISKFS-fs (dm-0): recovery complete [ 3138.401237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.693852] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3150.842146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3150.853243] Lustre: Skipped 41 previous similar messages [ 3150.971892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3150.975251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3159.446456] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3160.939152] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3169.515620] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:01:01 (1783141261) [ 3175.275134] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3176.751173] Lustre: Failing over lustre-MDT0000 [ 3177.050907] Lustre: server umount lustre-MDT0000 complete [ 3196.041627] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3196.049196] LDISKFS-fs (dm-0): recovery complete [ 3196.060245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.826653] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3201.602703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3201.604329] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3206.986664] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3208.269101] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3214.993709] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:01:47 (1783141307) [ 3220.362920] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3221.563484] Lustre: Failing over lustre-MDT0000 [ 3221.748202] Lustre: server umount lustre-MDT0000 complete [ 3239.404186] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3239.407055] LDISKFS-fs (dm-0): recovery complete [ 3239.414618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.580445] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3245.058240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3245.058246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3248.595604] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3249.594920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3254.830066] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 01:02:27 (1783141347) [ 3259.310986] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3260.659713] Lustre: Failing over lustre-MDT0000 [ 3260.857879] Lustre: server umount lustre-MDT0000 complete [ 3265.508561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.516898] Lustre: Skipped 45 previous similar messages [ 3279.249909] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3279.252068] LDISKFS-fs (dm-0): recovery complete [ 3279.259143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.563675] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3285.014796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3285.016967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3289.004258] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3289.902307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3295.387555] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 01:03:07 (1783141387) [ 3296.108720] Lustre: 98939:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ec529461-6aae-4389-8044-443dfe589617 at adminstrative request [ 3302.175299] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 01:03:14 (1783141394) [ 3306.585261] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3307.760359] Lustre: Failing over lustre-MDT0000 [ 3308.034387] Lustre: server umount lustre-MDT0000 complete [ 3316.284170] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3316.286509] LDISKFS-fs (dm-0): recovery complete [ 3316.292361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.520237] Lustre: lustre-MDT0000: Aborting client recovery [ 3316.522873] LustreError: 100221:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3316.527476] Lustre: 100253:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3316.532190] Lustre: 100253:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client ec529461-6aae-4389-8044-443dfe589617@ [ 3316.540823] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3316.545244] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3316.551559] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3316.586932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1480 to 0x280000401:1505) [ 3316.593488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 3319.569366] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3321.830033] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3336.508646] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 01:03:48 (1783141428) [ 3342.439550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3343.719985] Lustre: Failing over lustre-MDT0000 [ 3343.864146] Lustre: server umount lustre-MDT0000 complete [ 3352.693569] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3352.695502] LDISKFS-fs (dm-0): recovery complete [ 3352.701690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.751758] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.761588] LustreError: Skipped 11 previous similar messages [ 3352.914462] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3352.942158] Lustre: lustre-MDT0000: Aborting client recovery [ 3352.945234] LustreError: 102082:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3352.956595] Lustre: 102116:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3352.967083] Lustre: 102116:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3352.973755] Lustre: 102116:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3352.980015] Lustre: 102116:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3352.992433] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3352.996672] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3353.004988] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3353.042773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3353.043408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3355.874318] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3358.195563] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3367.977609] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3367.981289] Lustre: Skipped 1 previous similar message [ 3378.680396] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 01:04:28 (1783141468) [ 3389.216282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3391.877263] Lustre: Failing over lustre-MDT0000 [ 3392.488302] Lustre: server umount lustre-MDT0000 complete [ 3394.019575] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3394.040068] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 265 previous similar messages [ 3410.399247] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783141487/real 1783141487] req@ffff9dbe75e2ca80 x1869755906306048/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783141503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3410.409473] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 3412.511906] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3412.514465] LDISKFS-fs (dm-0): recovery complete [ 3412.531881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.640675] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe75ae4380 x1869755906314496/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3420.672491] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 3421.057647] Lustre: lustre-MDT0000: Aborting client recovery [ 3421.059713] LustreError: 103953:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3421.067144] Lustre: 103987:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3421.076639] Lustre: 103987:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3421.084313] Lustre: 103987:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3421.092199] Lustre: 103987:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3421.096920] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3421.103633] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3421.122596] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3421.187488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 3421.203825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 3426.042149] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3426.286395] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3451.133621] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 01:05:42 (1783141542) [ 3452.421649] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3452.423956] LustreError: 19435:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe4754f480 x1869755882676224/t201863462916(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:676/0 lens 512/456 e 0 to 0 dl 1783141556 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3456.117191] Lustre: Failing over lustre-MDT0000 [ 3456.203153] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 3456.325632] Lustre: server umount lustre-MDT0000 complete [ 3464.723554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.959577] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3464.961799] Lustre: lustre-MDT0000: Aborting client recovery [ 3464.962174] Lustre: Skipped 17 previous similar messages [ 3464.965203] LustreError: 105279:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3464.970413] Lustre: 105311:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3464.977510] Lustre: 105311:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3464.981842] Lustre: 105311:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3464.989992] Lustre: 105311:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3464.996442] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3465.002802] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3465.015567] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3465.056252] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 3465.075085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 3470.209933] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3470.310406] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3491.817540] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3493.664449] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 01:06:25 (1783141585) [ 3501.220971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3504.448579] Lustre: Failing over lustre-MDT0000 [ 3504.823114] Lustre: server umount lustre-MDT0000 complete [ 3519.951219] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3519.959064] LDISKFS-fs (dm-0): recovery complete [ 3519.976620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.512958] Lustre: lustre-MDT0000: Aborting client recovery [ 3520.516711] LustreError: 107242:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3520.528787] Lustre: 107276:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3520.535522] Lustre: 107276:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 3520.540445] Lustre: 107276:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3520.547645] Lustre: 107276:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3520.552389] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3520.559231] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3520.571970] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3520.680272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 3520.682579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 3525.382556] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3525.604512] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3548.895473] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 01:07:20 (1783141640) [ 3588.193563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3590.600973] Lustre: Failing over lustre-MDT0000 [ 3590.944570] Lustre: server umount lustre-MDT0000 complete [ 3613.593722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3613.595620] LDISKFS-fs (dm-0): recovery complete [ 3613.613492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.796472] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b07a7f9e [ 3618.804166] Lustre: Skipped 1 previous similar message [ 3620.001570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3620.004944] Lustre: Skipped 10 previous similar messages [ 3624.251426] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3624.549164] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3624.553802] Lustre: Skipped 10 previous similar messages [ 3624.592392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3624.594581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3634.685655] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3636.909388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3660.500615] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 01:09:12 (1783141752) [ 3689.873658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3700.614859] Lustre: Failing over lustre-MDT0000 [ 3700.848376] Lustre: server umount lustre-MDT0000 complete [ 3723.140157] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3723.145221] LDISKFS-fs (dm-0): recovery complete [ 3723.153955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.100237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.104132] Lustre: Skipped 11 previous similar messages [ 3730.999613] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3735.999509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3736.000674] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3742.392847] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3744.416522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3764.738779] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 01:10:56 (1783141856) [ 3766.748739] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3767.868953] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3767.877865] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3767.892529] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3767.903202] Lustre: Skipped 44 previous similar messages [ 3774.714709] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 01:11:06 (1783141866) [ 3801.855191] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3813.096785] Lustre: Failing over lustre-OST0000 [ 3813.178960] Lustre: server umount lustre-OST0000 complete [ 3837.035516] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3837.037963] LDISKFS-fs (dm-2): recovery complete [ 3837.043782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.179066] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3903.050758] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 01:13:14 (1783141994) [ 3911.476871] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3914.115283] Lustre: Failing over lustre-MDT0000 [ 3914.208933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.210832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3914.226487] Lustre: Skipped 33 previous similar messages [ 3914.233581] Lustre: Skipped 1 previous similar message [ 3914.412413] Lustre: server umount lustre-MDT0000 complete [ 3936.616993] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3936.629317] LDISKFS-fs (dm-0): recovery complete [ 3936.644650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.441050] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbd45be3100 x1869755907015936/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3945.464957] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 11 previous similar messages [ 3951.246960] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3951.248149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3951.283781] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3961.551795] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3963.920401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3967.455469] LustreError: 116691:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3967.457715] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3968.480186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3983.262346] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 01:14:35 (1783142075) [ 3988.255727] LustreError: 19435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3993.567137] LustreError: 19435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3993.574205] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 3993.620225] LustreError: 8433:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 3995.312081] LustreError: 9339:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4000.735159] LustreError: 9339:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4000.740144] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4002.614753] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4007.903338] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4007.911512] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4009.871689] LustreError: 19435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4015.071699] LustreError: 19435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4017.240322] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4022.751251] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4022.768367] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4022.782904] Lustre: Skipped 1 previous similar message [ 4032.275678] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4032.283507] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4037.599151] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4037.602601] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4044.767250] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4044.777679] Lustre: Skipped 2 previous similar messages [ 4053.877888] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4053.880542] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4059.104524] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4059.111386] LustreError: 8457:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4067.029750] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 01:15:59 (1783142159) [ 4068.819101] LustreError: 19435:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4078.249490] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4083.367266] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4088.498429] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4091.669305] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4098.720173] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4098.725234] Lustre: Skipped 1 previous similar message [ 4108.855211] LustreError: 19435:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4108.873986] Lustre: 19435:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9dbd46895500 x1869755885343104/t0(0) o38->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:0/0 lens 520/416 e 0 to 0 dl 1783142182 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4108.963229] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4108.971958] Lustre: Skipped 3 previous similar messages [ 4108.978656] LustreError: 6516:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4134.563533] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4134.568665] Lustre: Skipped 1 previous similar message [ 4149.031835] LustreError: 6516:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4149.042354] Lustre: 6516:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9dbe75e35880 x1869755885347584/t0(0) o38->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:0/0 lens 520/416 e 0 to 0 dl 1783142222 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4149.922277] LustreError: 9339:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4175.521218] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4175.542911] Lustre: Skipped 3 previous similar messages [ 4190.015283] LustreError: 9339:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4190.020692] Lustre: 9339:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9dbe44284700 x1869755885350784/t0(0) o38->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:0/0 lens 520/416 e 0 to 0 dl 1783142263 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4190.889588] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4190.900644] Lustre: Skipped 1 previous similar message [ 4190.906301] LustreError: 6515:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4216.479097] Lustre: lustre-MDT0000: Export ffff9dbe46a6b000 already connecting from 192.168.202.53@tcp [ 4216.484536] Lustre: Skipped 3 previous similar messages [ 4230.983153] LustreError: 6515:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4230.988799] Lustre: 6515:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9dbe80f3d180 x1869755885353984/t0(0) o38->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:0/0 lens 520/416 e 0 to 0 dl 1783142304 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4231.861269] LustreError: 6516:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4271.911166] LustreError: 6516:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4271.917734] Lustre: 6516:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9dbe442bf800 x1869755885357184/t0(0) o38->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:0/0 lens 520/416 e 0 to 0 dl 1783142345 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4272.801622] LustreError: 9339:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4295.991977] LustreError: 9339:0:(ldlm_lib.c:1422:target_handle_connect()) cfs_fail_timeout interrupted [ 4296.010068] Lustre: 9339:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/3s); client may timeout req@ffff9dbe79c60380 x1869755885360256/t0(0) o38->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:0/0 lens 520/416 e 0 to 0 dl 1783142386 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4301.832182] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 01:19:53 (1783142393) [ 4309.804937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4313.523666] Lustre: Failing over lustre-MDT0000 [ 4313.746757] Lustre: server umount lustre-MDT0000 complete [ 4313.770338] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.781188] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 172 previous similar messages [ 4315.104092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4315.112692] LustreError: Skipped 6 previous similar messages [ 4327.242968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4327.247347] LDISKFS-fs (dm-0): recovery complete [ 4327.255857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.475254] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.484793] LustreError: Skipped 6 previous similar messages [ 4327.680342] Lustre: *** cfs_fail_loc=712, val=0*** [ 4327.691848] LustreError: 43916:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9dbd436eb800 x1869755907212032/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4327.771969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.781160] Lustre: Skipped 2 previous similar messages [ 4327.852074] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4327.853379] Lustre: lustre-MDT0000: Aborting client recovery [ 4327.864640] Lustre: Skipped 9 previous similar messages [ 4327.869600] LustreError: 121548:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4327.903234] Lustre: 121580:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4327.917454] Lustre: 121580:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages [ 4327.929578] Lustre: 121580:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4327.944049] Lustre: 121580:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4327.953196] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4327.992263] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4328.022696] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4328.103987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4328.112710] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4332.361748] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4333.037075] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4348.030709] Lustre: Failing over lustre-MDT0000 [ 4348.221513] Lustre: server umount lustre-MDT0000 complete [ 4367.441832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.116969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4371.124089] Lustre: Skipped 3 previous similar messages [ 4372.974899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4372.981692] Lustre: Skipped 11 previous similar messages [ 4373.017098] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4373.026743] Lustre: Skipped 3 previous similar messages [ 4373.059456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4373.061518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4373.302260] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4383.134399] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4384.813898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4392.536040] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 01:21:24 (1783142484) [ 4392.690267] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 4392.696787] Lustre: Skipped 2 previous similar messages [ 4401.562914] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 01:21:33 (1783142493) [ 4402.640539] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4402.644334] LustreError: 6524:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe6f2b4e00 x1869755885421440/t0(0) o700->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:116/0 lens 264/248 e 0 to 0 dl 1783142506 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4421.625381] Lustre: Failing over lustre-MDT0000 [ 4421.858917] Lustre: server umount lustre-MDT0000 complete [ 4424.173989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4424.181071] LustreError: Skipped 1 previous similar message [ 4439.366494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.537257] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xe13102b9b07dcb1e to 0xe13102b9b07dcfbd [ 4439.553808] LustreError: 124261:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.153@tcp: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff9dbe75b9f500) refcount nonzero (3) after lock cleanup; forcing cleanup. [ 4439.713708] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (not set up) [ 4439.729485] Lustre: Skipped 2 previous similar messages [ 4443.719543] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4445.212124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4445.218309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4451.564502] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4452.834758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4462.320624] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:22:34 (1783142554) [ 4464.731453] Lustre: Failing over lustre-OST0000 [ 4467.042335] Lustre: server umount lustre-OST0000 complete [ 4483.941845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.085650] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4497.591327] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4498.729204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4569.775815] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:24:21 (1783142661) [ 4576.458706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4579.050271] Lustre: Failing over lustre-MDT0000 [ 4579.284959] Lustre: server umount lustre-MDT0000 complete [ 4581.367815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.389865] Lustre: Skipped 18 previous similar messages [ 4597.189207] Lustre: 3641:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783142674/real 1783142674] req@ffff9dbe79cfe680 x1869755907358208/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783142690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4597.208882] Lustre: 3641:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4601.054249] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4601.062543] LDISKFS-fs (dm-0): recovery complete [ 4601.077776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.457177] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe44885500 x1869755907366784/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4611.693863] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4613.657134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4613.657640] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4683.203938] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 01:26:15 (1783142775) [ 4684.792887] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4684.797559] Lustre: Skipped 3 previous similar messages [ 4696.253315] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 01:26:28 (1783142788) [ 4698.886637] Lustre: Failing over lustre-MDT0000 [ 4699.104840] Lustre: server umount lustre-MDT0000 complete [ 4716.756817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.069200] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4732.396595] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4732.401036] LustreError: 129298:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe60301f80 x1869755885548032/t0(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:486/0 lens 328/344 e 0 to 0 dl 1783142876 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4787.358122] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 4787.387710] Lustre: 129298:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4787.438422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 4787.438656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 4791.976740] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4793.305517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4800.478428] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 01:28:12 (1783142892) [ 4802.285926] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4809.478834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4811.094239] Lustre: Failing over lustre-MDT0000 [ 4811.269791] Lustre: server umount lustre-MDT0000 complete [ 4813.284539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4831.287861] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4831.290790] LDISKFS-fs (dm-0): recovery complete [ 4831.296402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.520460] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4845.580392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 4845.581908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4850.376699] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4851.516314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4858.397589] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 01:29:10 (1783142950) [ 4859.463869] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4865.878291] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4867.223105] Lustre: Failing over lustre-MDT0000 [ 4867.373494] Lustre: server umount lustre-MDT0000 complete [ 4886.616231] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4886.618827] LDISKFS-fs (dm-0): recovery complete [ 4886.625442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.040366] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4892.206864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4892.206991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3169) [ 4899.065542] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4900.825479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4908.275905] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 01:30:00 (1783143000) [ 4909.158702] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4917.041835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4918.688621] Lustre: Failing over lustre-MDT0000 [ 4918.990193] Lustre: server umount lustre-MDT0000 complete [ 4922.850617] LustreError: 7828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.870377] LustreError: 7828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 190 previous similar messages [ 4938.337806] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4938.339957] LDISKFS-fs (dm-0): recovery complete [ 4938.350135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.428817] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4938.437415] LustreError: Skipped 7 previous similar messages [ 4938.625776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.629414] Lustre: Skipped 7 previous similar messages [ 4938.656137] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4938.660817] Lustre: Skipped 9 previous similar messages [ 4942.118294] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4943.920839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3201) [ 4943.926052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4952.200441] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 01:30:44 (1783143044) [ 4954.022291] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4954.024244] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4954.028787] LustreError: 6518:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe75e3e300 x1869755885605760/t257698037777(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:668/0 lens 392/456 e 0 to 0 dl 1783143058 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4955.955600] Lustre: Failing over lustre-MDT0000 [ 4956.118083] Lustre: server umount lustre-MDT0000 complete [ 4972.610869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.264951] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4975.282394] Lustre: Skipped 7 previous similar messages [ 4976.432911] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4978.147991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4978.155309] Lustre: Skipped 32 previous similar messages [ 4978.180528] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4978.187352] Lustre: Skipped 7 previous similar messages [ 4978.201062] Lustre: 42539:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe6f367800 x1869755885605760/t257698037777(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:692/0 lens 392/456 e 0 to 0 dl 1783143082 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4978.226585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 4978.236501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 4982.780985] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4983.965876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4990.564634] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 01:31:22 (1783143082) [ 4991.527328] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4991.530332] LustreError: 6515:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe6f306300 x1869755885620352/t261993005072(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:749/0 lens 504/448 e 0 to 0 dl 1783143139 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4997.856390] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4999.232695] Lustre: Failing over lustre-MDT0000 [ 4999.356766] Lustre: server umount lustre-MDT0000 complete [ 5018.246906] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5018.248775] LDISKFS-fs (dm-0): recovery complete [ 5018.255871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.795060] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5023.754415] Lustre: 9339:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe80da2680 x1869755885620352/t261993005072(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:26/0 lens 504/2880 e 0 to 0 dl 1783143171 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5023.767299] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 5023.767502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 5028.573698] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5029.477342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5035.943814] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 01:32:08 (1783143128) [ 5036.700694] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5036.704821] LustreError: 6516:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe440fa680 x1869755885634688/t266287972368(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:39/0 lens 504/448 e 0 to 0 dl 1783143184 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5038.266744] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5043.246897] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5044.683376] Lustre: Failing over lustre-MDT0000 [ 5044.867751] Lustre: server umount lustre-MDT0000 complete [ 5063.645140] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5063.647350] LDISKFS-fs (dm-0): recovery complete [ 5063.656486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.737827] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5069.309184] Lustre: 9339:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe4481dc00 x1869755885634688/t266287972368(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:72/0 lens 504/2880 e 0 to 0 dl 1783143217 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5069.318186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3265) [ 5069.318279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 5069.324240] Lustre: 9339:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5076.270103] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 01:32:48 (1783143168) [ 5077.001443] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5077.004299] Lustre: Skipped 1 previous similar message [ 5077.009107] LustreError: 9339:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe44685500 x1869755885648512/t270582939664(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:80/0 lens 504/448 e 0 to 0 dl 1783143225 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5077.024703] LustreError: 9339:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message [ 5078.524340] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5078.527826] Lustre: Skipped 1 previous similar message [ 5083.841511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5085.066912] Lustre: Failing over lustre-MDT0000 [ 5085.210891] Lustre: server umount lustre-MDT0000 complete [ 5104.113811] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5104.116689] LDISKFS-fs (dm-0): recovery complete [ 5104.122635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.198120] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5109.776696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3297) [ 5109.776879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 5109.781059] Lustre: 6516:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe6f304a80 x1869755885648512/t270582939664(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:112/0 lens 504/2880 e 0 to 0 dl 1783143257 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5116.505932] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 01:33:28 (1783143208) [ 5117.263513] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5118.792886] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5118.796465] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5118.799057] LustreError: 42539:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe44750380 x1869755885662336/t274877906960(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:78/0 lens 392/456 e 0 to 0 dl 1783143223 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5124.065664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5125.108064] Lustre: Failing over lustre-MDT0000 [ 5125.275766] Lustre: server umount lustre-MDT0000 complete [ 5142.875723] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5142.877435] LDISKFS-fs (dm-0): recovery complete [ 5142.883501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.809406] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5148.178304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 5148.183358] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe79ce5880 x1869755885662336/t274877906960(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:107/0 lens 392/456 e 0 to 0 dl 1783143252 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5148.184047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3361) [ 5155.001028] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 01:34:07 (1783143247) [ 5155.653918] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5155.657165] LustreError: 6517:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe475d5c00 x1869755885672704/t279172874255(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:158/0 lens 664/608 e 0 to 0 dl 1783143303 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5215.393751] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnecting [ 5215.408222] Lustre: Skipped 1 previous similar message [ 5215.445078] Lustre: 6516:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbd436ebb80 x1869755885672704/t279172874255(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:218/0 lens 664/3488 e 0 to 0 dl 1783143363 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5219.842596] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 01:35:12 (1783143312) [ 5224.905182] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5226.547853] Lustre: Failing over lustre-MDT0000 [ 5226.737359] Lustre: server umount lustre-MDT0000 complete [ 5230.048992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5230.055380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.057613] LustreError: Skipped 3 previous similar messages [ 5230.075362] Lustre: Skipped 42 previous similar messages [ 5245.747246] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5245.748557] LDISKFS-fs (dm-0): recovery complete [ 5245.756342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.096533] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5251.105508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3361) [ 5251.108186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 5255.723273] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5256.927663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5273.328162] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 01:36:05 (1783143365) [ 5278.949888] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5280.356977] Lustre: Failing over lustre-MDT0000 [ 5280.523229] Lustre: server umount lustre-MDT0000 complete [ 5297.121207] Lustre: 3642:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783143374/real 1783143374] req@ffff9dbe44695180 x1869755907765376/t0(0) o400->MGC192.168.202.153@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783143390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5297.135360] Lustre: 3642:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5298.708631] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5298.710682] LDISKFS-fs (dm-0): recovery complete [ 5298.717338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.360617] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe447cdf80 x1869755907772544/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5310.133514] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5313.061148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 5313.061336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 5316.769863] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5317.905727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5322.385364] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5330.763438] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 01:37:02 (1783143422) [ 5359.617578] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5360.629385] Lustre: Failing over lustre-MDT0000 [ 5360.914849] Lustre: server umount lustre-MDT0000 complete [ 5377.807679] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5377.810204] LDISKFS-fs (dm-0): recovery complete [ 5377.815297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.088243] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5383.304675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5383.308236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5386.240799] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5387.223752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5417.615190] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 01:38:30 (1783143510) [ 5422.837636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5423.890594] Lustre: Failing over lustre-MDT0000 [ 5424.079827] Lustre: server umount lustre-MDT0000 complete [ 5441.184432] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5441.186561] LDISKFS-fs (dm-0): recovery complete [ 5441.193220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.872781] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5455.920614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 5455.920812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 5458.698904] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5459.539456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5465.176311] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5466.081277] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5469.784071] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 01:39:22 (1783143562) [ 5476.320588] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5477.486958] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5477.489634] Lustre: Skipped 1 previous similar message [ 5477.492210] LustreError: 6517:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe80f0f480 x1869755888462080/t296352743435(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:480/0 lens 66040/440 e 0 to 0 dl 1783143625 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5536.947801] Lustre: 6517:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe75292050 x1869755888462080/t296352743435(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:540/0 lens 66040/440 e 0 to 0 dl 1783143685 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5542.081609] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5543.136568] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 01:40:35 (1783143635) [ 5551.059476] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5552.781779] Lustre: Failing over lustre-MDT0000 [ 5552.958388] Lustre: server umount lustre-MDT0000 complete [ 5553.120220] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5553.130342] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 196 previous similar messages [ 5569.393905] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5569.396045] LDISKFS-fs (dm-0): recovery complete [ 5569.407674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.472611] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5569.476172] LustreError: Skipped 9 previous similar messages [ 5569.581657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.592060] Lustre: Skipped 9 previous similar messages [ 5569.609640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5569.613134] Lustre: Skipped 9 previous similar messages [ 5571.715863] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5575.173184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 5575.177348] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 5578.043155] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5578.860388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5584.911258] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 01:41:17 (1783143677) [ 5598.453098] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5606.460483] Lustre: Failing over lustre-OST0000 [ 5606.555081] Lustre: server umount lustre-OST0000 complete [ 5623.953061] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5623.954842] LDISKFS-fs (dm-2): recovery complete [ 5623.961712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5625.448606] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5625.452393] Lustre: Skipped 9 previous similar messages [ 5625.949494] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5625.949536] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5625.952209] Lustre: Skipped 39 previous similar messages [ 5625.960314] Lustre: Skipped 9 previous similar messages [ 5627.212599] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5639.468494] Lustre: Failing over lustre-OST0000 [ 5639.548654] Lustre: server umount lustre-OST0000 complete [ 5653.754175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5656.704394] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5661.712404] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5662.439940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5699.025227] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 01:43:11 (1783143791) [ 5700.534977] Lustre: Failing over lustre-MDT0000 [ 5700.554325] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143793 with bad export cookie 16226753928946152025 [ 5700.576136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5700.578798] Lustre: Skipped 2 previous similar messages [ 5700.734983] Lustre: server umount lustre-MDT0000 complete [ 5715.258953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.826381] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5720.582584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 5720.583825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 5730.111426] Lustre: Failing over lustre-MDT0000 [ 5730.278512] Lustre: server umount lustre-MDT0000 complete [ 5744.578878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.389068] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5749.747858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 5749.748144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 5752.113232] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5752.892369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5757.627535] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 01:44:10 (1783143850) [ 5769.795086] Lustre: Failing over lustre-OST0000 [ 5769.847360] Lustre: server umount lustre-OST0000 complete [ 5783.460066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5785.865921] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5790.425927] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5791.249390] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5796.871605] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 01:44:49 (1783143889) [ 5797.797947] Lustre: Failing over lustre-MDT0000 [ 5798.050221] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 5798.056284] Lustre: Skipped 4 previous similar messages [ 5798.100868] Lustre: server umount lustre-MDT0000 complete [ 5802.982716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.047635] Lustre: *** cfs_fail_loc=605, val=0*** [ 5803.048965] LustreError: 164741:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0a89b60 failed: rc = -95 [ 5803.052841] LustreError: 164741:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5803.055472] LustreError: 164741:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 5803.059359] LustreError: 164741:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5803.062492] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5803.064931] LustreError: 164741:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5803.072059] Lustre: server umount lustre-MDT0000 complete [ 5803.074361] LustreError: 164741:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5807.008964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.099965] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5812.227066] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5812.227066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5812.673377] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 01:45:05 (1783143905) [ 5816.043270] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5817.410461] Lustre: Failing over lustre-MDT0000 [ 5817.593451] Lustre: server umount lustre-MDT0000 complete [ 5832.550286] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5832.552384] LDISKFS-fs (dm-0): recovery complete [ 5832.555873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.887879] Lustre: *** cfs_fail_loc=707, val=0*** [ 5834.393560] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5894.311488] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5894.328238] Lustre: 166909:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5894.333196] Lustre: 166909:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message [ 5894.518419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 5894.518647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 5896.894121] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5897.546182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5901.659563] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 01:46:34 (1783143994) [ 5926.073388] LustreError: 19435:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe4754f800 x1869755889346560/t0(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:130/0 lens 664/0 e 0 to 0 dl 1783144030 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5926.083165] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5937.111100] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5937.119964] LustreError: 42539:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe75e2c000 x1869755889347584/t0(0) o35->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:141/0 lens 392/0 e 0 to 0 dl 1783144041 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5942.753350] LustreError: 8431:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe75864380 x1869755908971648/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:146/0 lens 544/0 e 0 to 0 dl 1783144046 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 5942.761898] LustreError: 8431:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 5949.585773] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 01:47:22 (1783144042) [ 5974.749265] LustreError: 42744:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5985.775106] LustreError: 42744:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5990.294855] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 01:48:02 (1783144082) [ 6014.872291] LustreError: 19435:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe75e3c700 x1869755889370112/t0(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:219/0 lens 576/0 e 0 to 0 dl 1783144119 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6014.881475] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6019.888497] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6019.898757] LustreError: 6516:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe75e3f800 x1869755889370880/t0(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:224/0 lens 664/0 e 0 to 0 dl 1783144124 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6031.063067] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6031.068397] LustreError: 8457:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbd4268a680 x1869755889392384/t0(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:274/0 lens 664/0 e 0 to 0 dl 1783144174 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6031.079753] LustreError: 8457:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 6044.352691] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 01:48:56 (1783144136) [ 6130.614375] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 01:50:23 (1783144223) [ 6154.933618] LustreError: 19435:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe75e3ca80 x1869755889446528/t0(0) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:359/0 lens 576/0 e 0 to 0 dl 1783144259 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6154.943300] LustreError: 19435:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 6154.947586] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6154.951455] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 6155.375163] LustreError: 19435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6171.315211] LustreError: 8457:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6171.320853] LustreError: 8457:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 6171.743098] LustreError: 8457:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6171.746280] LustreError: 8457:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 6187.249282] LustreError: 6517:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9dbe447a6300 x1869755889465344/t0(0) o36->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:391/0 lens 504/0 e 0 to 0 dl 1783144291 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 6187.256513] LustreError: 6517:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 6199.723462] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 01:51:32 (1783144292) [ 6225.016607] Lustre: DEBUG MARKER: phase 2 [ 6228.482271] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 01:52:01 (1783144321) [ 6299.182405] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 01:53:11 (1783144391) [ 6299.806611] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6300.437840] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 01:53:13 (1783144393) [ 6302.408427] Lustre: DEBUG MARKER: Started rundbench load pid=135276 ... [ 6307.193580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6309.016345] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6309.904935] Lustre: Failing over lustre-MDT0000 [ 6310.057582] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 6310.193404] Lustre: server umount lustre-MDT0000 complete [ 6313.952084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.953745] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6313.956169] Lustre: Skipped 40 previous similar messages [ 6313.964031] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 90 previous similar messages [ 6325.286422] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6325.288042] LDISKFS-fs (dm-0): recovery complete [ 6325.292446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6325.355694] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6325.361201] LustreError: Skipped 4 previous similar messages [ 6325.434211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.437679] Lustre: Skipped 7 previous similar messages [ 6325.454538] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6325.457419] Lustre: Skipped 7 previous similar messages [ 6326.561014] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6326.565848] Lustre: Skipped 6 previous similar messages [ 6327.065435] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6330.852734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6330.856969] Lustre: Skipped 21 previous similar messages [ 6330.868804] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6330.873031] Lustre: Skipped 6 previous similar messages [ 6330.890460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 6330.890658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 6333.089676] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6333.907353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6339.700911] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6341.328698] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6342.090633] Lustre: Failing over lustre-MDT0001 [ 6342.287595] Lustre: server umount lustre-MDT0001 complete [ 6346.209976] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6346.212700] LustreError: Skipped 7 previous similar messages [ 6356.624578] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 6356.626405] LDISKFS-fs (dm-1): recovery complete [ 6356.630663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.110093] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6362.265986] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:629 to 0x2c0000400:673) [ 6362.266036] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:647 to 0x280000400:673) [ 6364.337692] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6365.021650] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6370.519145] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6372.280284] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6373.111175] Lustre: Failing over lustre-MDT0000 [ 6373.387686] Lustre: server umount lustre-MDT0000 complete [ 6388.249566] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6388.251203] LDISKFS-fs (dm-0): recovery complete [ 6388.256415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.065412] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6393.853278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 6393.853733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 6395.993642] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6396.605400] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6402.018325] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6403.665213] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 6404.390747] Lustre: Failing over lustre-MDT0001 [ 6404.537164] Lustre: server umount lustre-MDT0001 complete [ 6419.157273] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6419.159112] LDISKFS-fs (dm-1): recovery complete [ 6419.164821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.740643] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6424.874161] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:929 to 0x2c0000400:961) [ 6424.874423] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:930 to 0x280000400:961) [ 6426.998086] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6427.637752] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6448.612063] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 01:55:41 (1783144541) [ 6571.482649] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6582.302153] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6582.941230] Lustre: Failing over lustre-MDT0000 [ 6583.263499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6583.265996] Lustre: Skipped 6 previous similar messages [ 6589.006836] Lustre: server umount lustre-MDT0000 complete [ 6603.063095] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6603.065442] LDISKFS-fs (dm-0): recovery complete [ 6603.069515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6604.476554] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6610.386910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6289 to 0x280000401:6305) [ 6610.391649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6225 to 0x2c0000401:6241) [ 6612.236060] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6612.813787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6737.003654] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6747.820766] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 6748.483784] Lustre: Failing over lustre-MDT0001 [ 6748.731292] Lustre: server umount lustre-MDT0001 complete [ 6763.391361] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6763.392804] LDISKFS-fs (dm-1): recovery complete [ 6763.397449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.965456] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6770.771736] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3189 to 0x280000400:3233) [ 6770.772079] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3190 to 0x2c0000400:3233) [ 6772.697386] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6773.301248] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6784.941529] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 02:01:17 (1783144877) [ 6907.873659] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6918.645959] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6919.324825] Lustre: Failing over lustre-MDT0000 [ 6919.325070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.331162] Lustre: Skipped 18 previous similar messages [ 6919.332764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.335408] Lustre: Skipped 6 previous similar messages [ 6919.721497] Lustre: server umount lustre-MDT0000 complete [ 6922.208670] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.218359] LustreError: 9339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 73 previous similar messages [ 6933.844041] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6933.846148] LDISKFS-fs (dm-0): recovery complete [ 6933.849729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.894876] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.898122] LustreError: Skipped 2 previous similar messages [ 6933.976836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.979099] Lustre: Skipped 5 previous similar messages [ 6933.989769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6933.992798] Lustre: Skipped 5 previous similar messages [ 6935.327298] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6937.757563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6937.761276] Lustre: Skipped 5 previous similar messages [ 6939.105351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6939.107307] Lustre: Skipped 20 previous similar messages [ 6940.107250] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6940.110424] Lustre: Skipped 5 previous similar messages [ 6940.129329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7725 to 0x280000401:7745) [ 6940.129903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7660 to 0x2c0000401:7681) [ 6942.068244] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6942.767267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7066.792968] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7077.506845] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 7078.096858] Lustre: Failing over lustre-MDT0001 [ 7082.464183] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7082.466613] LustreError: Skipped 5 previous similar messages [ 7084.096231] Lustre: server umount lustre-MDT0001 complete [ 7098.150446] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7098.151511] LDISKFS-fs (dm-1): recovery complete [ 7098.155429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.450518] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7104.459459] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4071 to 0x2c0000400:4097) [ 7104.464672] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4070 to 0x280000400:4097) [ 7106.315577] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7106.875521] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7110.167714] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 02:06:42 (1783145202) [ 7233.035532] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7243.826620] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7244.421892] Lustre: Failing over lustre-MDT0001 [ 7244.593514] Lustre: server umount lustre-MDT0001 complete [ 7258.610236] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7258.611685] LDISKFS-fs (dm-1): recovery complete [ 7258.616687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.095332] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7264.244047] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4109 to 0x280000400:4129) [ 7264.244160] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4110 to 0x2c0000400:4129) [ 7266.118824] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7266.754186] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7390.952538] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7401.708460] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 7402.351047] Lustre: Failing over lustre-MDT0001 [ 7402.583714] Lustre: server umount lustre-MDT0001 complete [ 7416.729262] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7416.730684] LDISKFS-fs (dm-1): recovery complete [ 7416.735011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.860721] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7418.360383] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7421.945599] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4110 to 0x2c0000400:4161) [ 7421.945609] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4109 to 0x280000400:4161) [ 7424.155376] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7424.784995] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7428.462747] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 02:12:01 (1783145521) [ 7434.208361] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7435.691108] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7436.280877] Lustre: Failing over lustre-OST0000 [ 7436.318270] Lustre: server umount lustre-OST0000 complete [ 7450.375660] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7450.377642] LDISKFS-fs (dm-2): recovery complete [ 7450.381470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.445851] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7452.446918] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7456.333449] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7456.886149] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7465.666074] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7467.146039] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7467.745842] Lustre: Failing over lustre-OST0000 [ 7467.782325] Lustre: server umount lustre-OST0000 complete [ 7481.551220] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7481.553073] LDISKFS-fs (dm-2): recovery complete [ 7481.555772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7481.598778] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7483.326531] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7487.402162] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7488.056088] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7493.683584] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 02:13:06 (1783145586) [ 7616.729078] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7619.263642] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7630.025439] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7630.667912] Lustre: Failing over lustre-MDT0001 [ 7630.672535] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7630.675711] Lustre: Skipped 16 previous similar messages [ 7630.676925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7630.678672] Lustre: Skipped 8 previous similar messages [ 7630.806537] Lustre: server umount lustre-MDT0001 complete [ 7632.023971] Lustre: Failing over lustre-MDT0000 [ 7633.053950] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.060296] LustreError: 19435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 63 previous similar messages [ 7638.184488] Lustre: server umount lustre-MDT0000 complete [ 7653.169306] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7653.171386] LDISKFS-fs (dm-1): recovery complete [ 7653.175492] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7653.176902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.177879] LDISKFS-fs (dm-0): recovery complete [ 7653.185504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.231782] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7653.347844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7653.350878] Lustre: Skipped 2 previous similar messages [ 7653.369212] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7653.372571] Lustre: Skipped 5 previous similar messages [ 7653.532860] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7653.535941] Lustre: Skipped 5 previous similar messages [ 7654.965336] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7655.112855] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7655.521760] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7655.525136] Lustre: Skipped 16 previous similar messages [ 7656.422723] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7656.425923] Lustre: Skipped 5 previous similar messages [ 7656.440363] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4110 to 0x2c0000400:4193) [ 7656.440385] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4109 to 0x280000400:4193) [ 7664.158222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12831 to 0x2c0000401:12865) [ 7664.158291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12896 to 0x280000401:12929) [ 7666.043528] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7666.620901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7667.235728] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7791.501142] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7794.158285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7804.939952] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 7805.588521] Lustre: Failing over lustre-MDT0001 [ 7805.589406] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -19 [ 7805.592762] LustreError: Skipped 5 previous similar messages [ 7805.817639] Lustre: server umount lustre-MDT0001 complete [ 7807.060044] Lustre: Failing over lustre-MDT0000 [ 7813.248981] Lustre: server umount lustre-MDT0000 complete [ 7828.067904] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7828.071085] LDISKFS-fs (dm-1): recovery complete [ 7828.076110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.150074] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7828.151248] LDISKFS-fs (dm-0): recovery complete [ 7828.154635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.193402] LustreError: 201913:0:(llog.c:1646:llog_backup()) MGC192.168.202.153@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7828.194159] LustreError: 201912:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.153@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9dbe60716200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7828.197284] Lustre: 201913:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.153@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7829.876963] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7830.024331] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7837.871690] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4109 to 0x280000400:4225) [ 7837.871738] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4110 to 0x2c0000400:4225) [ 7847.039289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12896 to 0x280000401:12961) [ 7847.050404] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12831 to 0x2c0000401:12897) [ 7849.288928] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7849.920836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7850.492231] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7854.145824] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 02:19:06 (1783145946) [ 7856.849512] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7857.914467] Lustre: Failing over lustre-MDT0000 [ 7858.115074] Lustre: server umount lustre-MDT0000 complete [ 7872.238820] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7872.240826] LDISKFS-fs (dm-0): recovery complete [ 7872.245134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.671398] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7873.826618] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7888.038116] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7888.048332] Lustre: 201919:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbd4d4c0000 x1869755969175296/t335007513059(335007513059) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:582/0 lens 520/3488 e 0 to 0 dl 1783145992 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7888.079157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:12993) [ 7888.079332] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12831 to 0x2c0000401:12929) [ 7890.012397] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7890.521683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7893.913413] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 02:19:46 (1783145986) [ 7896.428413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7897.371371] Lustre: Failing over lustre-MDT0000 [ 7897.559066] Lustre: server umount lustre-MDT0000 complete [ 7911.505609] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7911.507338] LDISKFS-fs (dm-0): recovery complete [ 7911.509987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7912.969198] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7913.638955] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7913.640328] LustreError: 201920:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe44116680 x1869755969175296/t335007513059(335007513059) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:607/0 lens 520/664 e 0 to 0 dl 1783146017 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7928.997048] Lustre: lustre-MDT0000: Client ec529461-6aae-4389-8044-443dfe589617 (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7929.003163] Lustre: 203012:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbd4d2aad80 x1869755969175296/t335007513059(335007513059) o101->ec529461-6aae-4389-8044-443dfe589617@192.168.202.53@tcp:623/0 lens 520/3488 e 0 to 0 dl 1783146033 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7929.034170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13025) [ 7929.034171] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12831 to 0x2c0000401:12961) [ 7930.973880] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7931.526571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7934.877744] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 02:20:27 (1783146027) [ 7936.726029] Lustre: Failing over lustre-OST0000 [ 7936.773833] Lustre: server umount lustre-OST0000 complete [ 7937.976953] Lustre: Failing over lustre-MDT0000 [ 7938.075129] Lustre: server umount lustre-MDT0000 complete [ 7950.874052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7952.334942] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7953.064811] Lustre: lustre-MDT0000: Denying connection for new client 4e3b3008-aa74-4224-9540-0230f9cbcb5a (at 192.168.202.53@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 7956.469981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12831 to 0x2c0000401:12993) [ 7960.058283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7962.077894] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7965.156837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13057) [ 7966.190237] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 02:20:58 (1783146058) [ 7966.548158] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7966.550236] LustreError: 8441:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe444b3b80 x1869755955333504/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:660/0 lens 1312/4320 e 0 to 0 dl 1783146070 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7968.933456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7969.549839] Lustre: Failing over lustre-MDT0000 [ 7969.821169] Lustre: server umount lustre-MDT0000 complete [ 7982.047122] Lustre: 201946:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146059/real 1783146059] req@ffff9dbe444b3800 x1869755955333504/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1783146075 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7982.053948] Lustre: 201946:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7984.031401] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7984.033089] LDISKFS-fs (dm-0): recovery complete [ 7984.035996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.739494] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7989.242463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13025) [ 7989.242466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13089) [ 7991.098241] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7991.697189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7995.498387] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 02:21:28 (1783146088) [ 7998.455510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8001.181305] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8001.791199] Lustre: Failing over lustre-MDT0001 [ 8008.250478] Lustre: server umount lustre-MDT0001 complete [ 8022.330751] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8022.332972] LDISKFS-fs (dm-1): recovery complete [ 8022.337673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.732925] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8027.631391] Lustre: 201920:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbd4ee9a300 x1869755983051520/t38654708165(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:743/0 lens 560/2880 e 0 to 0 dl 1783146153 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8027.635876] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4236 to 0x280000400:4257) [ 8027.636408] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4236 to 0x2c0000400:4257) [ 8029.330144] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8029.891170] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8033.510613] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 02:22:06 (1783146126) [ 8033.885618] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8033.887041] Lustre: Skipped 1 previous similar message [ 8033.888348] LustreError: 8441:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe4468f800 x1869755955396480/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:728/0 lens 2520/4320 e 0 to 0 dl 1783146138 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8033.895117] LustreError: 8441:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message [ 8036.338686] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8038.871259] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8039.508119] Lustre: Failing over lustre-MDT0000 [ 8039.755845] Lustre: server umount lustre-MDT0000 complete [ 8053.993241] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8053.994456] LDISKFS-fs (dm-0): recovery complete [ 8053.997309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.435903] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8059.406839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13121) [ 8059.406853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13057) [ 8061.334880] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8061.886218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8063.927046] Lustre: Failing over lustre-MDT0001 [ 8064.017277] Lustre: server umount lustre-MDT0001 complete [ 8078.387780] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8078.390396] LDISKFS-fs (dm-1): recovery complete [ 8078.397422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8079.942305] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8083.981906] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4268 to 0x280000400:4289) [ 8083.981937] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4289) [ 8085.858093] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8086.452405] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8089.969484] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 02:23:02 (1783146182) [ 8095.869386] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8098.381055] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8099.123316] Lustre: Failing over lustre-MDT0000 [ 8099.458311] Lustre: server umount lustre-MDT0000 complete [ 8100.898865] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146194 with bad export cookie 16226753928955013430 [ 8100.900248] Lustre: Failing over lustre-MDT0001 [ 8100.902865] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8106.975760] LustreError: 218412:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9dbd42436d80 x1869755955458432/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8106.983704] LustreError: 218412:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8107.084748] Lustre: server umount lustre-MDT0001 complete [ 8121.968500] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8121.969889] LDISKFS-fs (dm-0): recovery complete [ 8121.974062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8122.024536] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8122.026433] LDISKFS-fs (dm-1): recovery complete [ 8122.030548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.855517] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbd42434700 x1869755955459200/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8127.663136] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8127.749404] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8131.128743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13153) [ 8131.128856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13089) [ 8131.146793] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4321) [ 8131.147902] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4321) [ 8131.154297] Lustre: 219526:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbd4d4c1880 x1869755983117952/t47244640324(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:95/0 lens 560/2880 e 0 to 0 dl 1783146260 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8132.941000] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8133.455785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8133.962091] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8137.511894] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 02:23:50 (1783146230) [ 8137.869405] LustreError: 220294:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe47abfb80 x1869755983149696/t51539607620(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:116/0 lens 560/448 e 0 to 0 dl 1783146281 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8137.880173] LustreError: 220294:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 1 previous similar message [ 8143.404901] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8144.017837] Lustre: Failing over lustre-MDT0000 [ 8144.265048] Lustre: server umount lustre-MDT0000 complete [ 8158.508119] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8158.509636] LDISKFS-fs (dm-0): recovery complete [ 8158.513069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8160.031710] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8163.828411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13121) [ 8163.828430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13185) [ 8165.693200] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8166.253571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8193.180887] Lustre: lustre-MDT0001: Client 4e3b3008-aa74-4224-9540-0230f9cbcb5a (at 192.168.202.53@tcp) reconnecting [ 8193.183327] Lustre: Skipped 1 previous similar message [ 8193.185636] Lustre: 219526:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe44116300 x1869755983149696/t51539607620(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:171/0 lens 560/2880 e 0 to 0 dl 1783146336 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8195.543627] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 02:24:48 (1783146288) [ 8195.899933] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8195.901569] Lustre: Skipped 2 previous similar messages [ 8198.442708] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8199.082970] Lustre: Failing over lustre-MDT0001 [ 8199.166322] Lustre: server umount lustre-MDT0001 complete [ 8213.643806] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8213.646168] LDISKFS-fs (dm-1): recovery complete [ 8213.652777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.244593] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8219.128186] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4342 to 0x2c0000400:4385) [ 8219.128242] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4342 to 0x280000400:4385) [ 8220.950824] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8221.487413] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8224.999948] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 02:25:17 (1783146317) [ 8230.851187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8233.418466] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8234.070034] Lustre: Failing over lustre-MDT0000 [ 8234.345396] Lustre: server umount lustre-MDT0000 complete [ 8234.464314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.464841] LustreError: 8441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8234.467936] Lustre: Skipped 51 previous similar messages [ 8234.472461] LustreError: 8441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 144 previous similar messages [ 8248.508456] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8248.510346] LDISKFS-fs (dm-0): recovery complete [ 8248.513926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.973704] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8253.953321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13153) [ 8253.953382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13217) [ 8255.869378] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8256.404434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8258.361232] Lustre: Failing over lustre-MDT0001 [ 8258.445195] Lustre: server umount lustre-MDT0001 complete [ 8272.690112] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8272.692067] LDISKFS-fs (dm-1): recovery complete [ 8272.697597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.806500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8272.808926] Lustre: Skipped 16 previous similar messages [ 8272.817552] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8272.820836] Lustre: Skipped 16 previous similar messages [ 8274.327321] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8274.719468] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8274.722838] Lustre: Skipped 16 previous similar messages [ 8277.984917] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8277.986756] Lustre: Skipped 55 previous similar messages [ 8277.991093] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8277.993740] Lustre: Skipped 16 previous similar messages [ 8278.007014] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4396 to 0x280000400:4417) [ 8278.007023] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4396 to 0x2c0000400:4417) [ 8279.881291] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8280.448513] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8284.055573] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 02:26:16 (1783146376) [ 8284.409785] LustreError: 221427:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe475d6680 x1869755983250176/t60129542212(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:223/0 lens 560/448 e 0 to 0 dl 1783146388 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8284.420628] LustreError: 221427:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 2 previous similar messages [ 8289.817482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8292.304211] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8292.950497] Lustre: Failing over lustre-MDT0000 [ 8293.225664] Lustre: server umount lustre-MDT0000 complete [ 8294.493876] Lustre: Failing over lustre-MDT0001 [ 8294.493906] LustreError: 43912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146387 with bad export cookie 16226753928955027157 [ 8294.494550] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8294.494557] LustreError: Skipped 9 previous similar messages [ 8294.505021] LustreError: 43912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8294.600585] Lustre: server umount lustre-MDT0001 complete [ 8309.554342] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8309.556444] LDISKFS-fs (dm-1): recovery complete [ 8309.557254] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8309.560970] LDISKFS-fs (dm-0): recovery complete [ 8309.561765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.564182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.652368] LustreError: 230667:0:(llog.c:1646:llog_backup()) MGC192.168.202.153@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8309.655092] Lustre: 230667:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.153@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8313.823328] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146391/real 1783146391] req@ffff9dbe6ba8b800 x1869755955623040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8313.831481] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8318.943406] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbd45d30700 x1869755955626240/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8320.511294] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8320.603881] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8325.167790] Lustre: 230696:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe44116a00 x1869755983250176/t60129542212(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:264/0 lens 560/2880 e 0 to 0 dl 1783146429 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8325.176441] Lustre: 230696:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 8325.179751] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4428 to 0x2c0000400:4449) [ 8325.180106] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4428 to 0x280000400:4449) [ 8325.194598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13249) [ 8325.194616] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13185) [ 8326.987518] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8327.527528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8328.020800] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8331.625678] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 02:27:04 (1783146424) [ 8334.460324] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8335.068572] Lustre: Failing over lustre-MDT0001 [ 8335.073235] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.53@tcp (stopping) [ 8335.076123] Lustre: Skipped 25 previous similar messages [ 8341.042227] Lustre: server umount lustre-MDT0001 complete [ 8355.150789] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8355.152378] LDISKFS-fs (dm-1): recovery complete [ 8355.157124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.620524] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8360.432909] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4481) [ 8360.432919] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4481) [ 8362.172864] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.691309] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8366.181186] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 02:27:38 (1783146458) [ 8369.043619] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8369.703703] Lustre: Failing over lustre-MDT0000 [ 8369.963415] Lustre: server umount lustre-MDT0000 complete [ 8383.996292] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8383.997669] LDISKFS-fs (dm-0): recovery complete [ 8384.000770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.508421] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8389.629323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13281) [ 8389.629322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13217) [ 8391.372643] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8391.897418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8395.453074] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 02:28:08 (1783146488) [ 8398.239125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8400.706567] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8401.369841] Lustre: Failing over lustre-MDT0000 [ 8401.615566] Lustre: server umount lustre-MDT0000 complete [ 8415.703248] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8415.704673] LDISKFS-fs (dm-0): recovery complete [ 8415.707340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.206474] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8420.864476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13249) [ 8420.864476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13313) [ 8422.721656] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8423.309557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8425.166920] Lustre: Failing over lustre-MDT0001 [ 8425.250377] Lustre: server umount lustre-MDT0001 complete [ 8438.955865] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8438.957050] LDISKFS-fs (dm-1): recovery complete [ 8438.960220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8440.242198] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8444.405338] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4513) [ 8444.405385] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4513) [ 8446.014942] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8446.504617] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8449.767723] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 02:29:02 (1783146542) [ 8452.357567] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8454.600629] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8455.168555] Lustre: Failing over lustre-MDT0000 [ 8455.463517] Lustre: server umount lustre-MDT0000 complete [ 8456.559624] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146549 with bad export cookie 16226753928955037566 [ 8456.560638] Lustre: Failing over lustre-MDT0001 [ 8456.562394] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8461.791842] LustreError: 240698:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9dbe7689a300 x1869755955761280/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8461.798277] LustreError: 240698:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8461.917488] Lustre: server umount lustre-MDT0001 complete [ 8477.056584] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8477.059256] LDISKFS-fs (dm-0): recovery complete [ 8477.060855] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8477.062871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.063920] LDISKFS-fs (dm-1): recovery complete [ 8477.074126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.247499] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9dbe7b189180 x1869755955762048/t0(0) o250->MGC192.168.202.153@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8481.477664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8481.480315] LustreError: Skipped 16 previous similar messages [ 8483.049143] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8483.157693] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8487.556986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13345) [ 8487.556995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13281) [ 8487.568706] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4545) [ 8487.568747] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4545) [ 8487.582782] Lustre: 241813:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9dbe75e20e00 x1869755983326592/t73014444034(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:451/0 lens 496/2888 e 0 to 0 dl 1783146616 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8487.589779] Lustre: 241813:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8489.414204] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8489.998740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8490.546410] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8494.016700] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 02:29:46 (1783146586) [ 8494.398411] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8494.400765] Lustre: Skipped 6 previous similar messages [ 8497.084870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8497.709289] Lustre: Failing over lustre-MDT0000 [ 8497.861905] Lustre: server umount lustre-MDT0000 complete [ 8511.865332] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8511.866674] LDISKFS-fs (dm-0): recovery complete [ 8511.870085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.372945] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8517.118902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13377) [ 8517.118922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13313) [ 8518.852240] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8519.412517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8549.533064] Lustre: lustre-MDT0001: Client 4e3b3008-aa74-4224-9540-0230f9cbcb5a (at 192.168.202.53@tcp) reconnecting [ 8549.535488] Lustre: Skipped 1 previous similar message [ 8551.874126] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 02:30:44 (1783146644) [ 8552.229607] LustreError: 241815:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ffff9dbe44114380 x1869755983356160/t77309411333(0) o36->4e3b3008-aa74-4224-9540-0230f9cbcb5a@192.168.202.53@tcp:491/0 lens 496/456 e 0 to 0 dl 1783146656 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8552.236317] LustreError: 241815:0:(ldlm_lib.c:3332:target_send_reply_msg()) Skipped 5 previous similar messages [ 8554.630371] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8555.251918] Lustre: Failing over lustre-MDT0001 [ 8555.347384] Lustre: server umount lustre-MDT0001 complete [ 8569.432352] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8569.434628] LDISKFS-fs (dm-1): recovery complete [ 8569.443775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8570.976943] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8574.962578] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4577) [ 8574.962587] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4577) [ 8576.724909] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8577.255611] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8580.705247] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 02:31:13 (1783146673) [ 8583.311710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8585.663501] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8586.257218] Lustre: Failing over lustre-MDT0000 [ 8586.504288] Lustre: server umount lustre-MDT0000 complete [ 8600.633325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8600.635142] LDISKFS-fs (dm-0): recovery complete [ 8600.638214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.112171] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8606.205321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13409) [ 8606.205337] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13345) [ 8608.013439] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8608.605338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8610.565531] Lustre: Failing over lustre-MDT0001 [ 8610.658370] Lustre: server umount lustre-MDT0001 complete [ 8624.873430] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8624.875070] LDISKFS-fs (dm-1): recovery complete [ 8624.880267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8626.446792] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8630.258216] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4609) [ 8630.258217] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4609) [ 8632.064776] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8632.611550] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8635.995401] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 02:32:08 (1783146728) [ 8638.912082] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8641.418181] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8642.052792] Lustre: Failing over lustre-MDT0000 [ 8642.166860] Lustre: server umount lustre-MDT0000 complete [ 8643.372594] LustreError: 184435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146736 with bad export cookie 16226753928955045112 [ 8643.374134] Lustre: Failing over lustre-MDT0001 [ 8643.376429] LustreError: 184435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8643.487770] Lustre: server umount lustre-MDT0001 complete [ 8658.641153] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8658.643692] LDISKFS-fs (dm-0): recovery complete [ 8658.646479] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8658.647265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8658.648176] LDISKFS-fs (dm-1): recovery complete [ 8658.658469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8660.959294] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783146738/real 1783146738] req@ffff9dbe81237100 x1869755955904384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783146754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8660.966507] Lustre: 3640:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 8670.959120] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8671.086652] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8672.442572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12995 to 0x2c0000401:13377) [ 8672.442573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12995 to 0x280000401:13441) [ 8677.554417] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4641) [ 8677.554483] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4641) [ 8679.411685] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8679.976155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8680.481340] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8684.113500] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 02:32:56 (1783146776) [ 8684.746628] Lustre: 254352:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4e3b3008-aa74-4224-9540-0230f9cbcb5a at adminstrative request [ 8689.549250] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 02:33:02 (1783146782) [ 8691.465086] Lustre: Failing over lustre-MDT0000 [ 8691.744156] Lustre: server umount lustre-MDT0000 complete [ 8704.645216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.175855] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8710.161410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13428 to 0x2c0000401:13473) [ 8710.161414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13493 to 0x280000401:13537) [ 8711.939766] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8712.535926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8715.822135] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 02:33:28 (1783146808) [ 8719.849306] Lustre: Failing over lustre-OST0000 [ 8719.909075] Lustre: server umount lustre-OST0000 complete [ 8732.726490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8734.785309] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8738.521607] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8739.072329] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8742.782800] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 02:33:55 (1783146835) [ 8744.066567] Lustre: Failing over lustre-MDT0000 [ 8744.327385] Lustre: server umount lustre-MDT0000 complete [ 8746.823298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.393329] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8749.167846] Lustre: lustre-MDT0000: Denying connection for new client d063271f-0a4b-48af-8510-377f4b22180c (at 192.168.202.53@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8749.178202] Lustre: Skipped 2 previous similar messages [ 8752.113501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13493 to 0x280000401:13569) [ 8752.113512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13428 to 0x2c0000401:13505) [ 8756.507450] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 02:34:09 (1783146849) [ 8759.335543] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8760.233911] Lustre: Failing over lustre-OST0000 [ 8760.272020] Lustre: server umount lustre-OST0000 complete [ 8774.221090] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8774.222385] LDISKFS-fs (dm-2): recovery complete [ 8774.226221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8776.238253] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8779.969148] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8780.508855] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8784.026201] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 02:34:36 (1783146876) [ 8786.769040] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8788.627440] Lustre: Failing over lustre-OST0000 [ 8788.663745] Lustre: server umount lustre-OST0000 complete [ 8802.816522] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8802.817966] LDISKFS-fs (dm-2): recovery complete [ 8802.820838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8804.659402] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.53@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:13571 extent [0-4194303]: client csum 2b1cf2e3, server csum e3d4f2fa [ 8804.924782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8808.726683] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8809.296746] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8812.662461] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 02:35:05 (1783146905) [ 8814.989281] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8817.365872] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8819.203992] Lustre: Failing over lustre-MDT0000 [ 8819.504592] Lustre: server umount lustre-MDT0000 complete [ 8820.673794] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146913 with bad export cookie 16226753928955071929 [ 8820.675881] Lustre: Failing over lustre-OST0000 [ 8820.677711] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8820.714944] Lustre: server umount lustre-OST0000 complete [ 8834.979453] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8834.980832] LDISKFS-fs (dm-0): recovery complete [ 8834.983284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8835.551606] LustreError: 253780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.557112] LustreError: 253780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 224 previous similar messages [ 8845.793168] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe13102b9b10a9cd8 [ 8847.201607] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8851.029237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13428 to 0x2c0000401:13537) [ 8861.667305] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8861.668691] LDISKFS-fs (dm-2): recovery complete [ 8861.672044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.734367] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8870.639605] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 02:36:03 (1783146963) [ 8875.421982] Lustre: Failing over lustre-OST0000 [ 8875.472825] Lustre: server umount lustre-OST0000 complete [ 8876.733787] Lustre: Failing over lustre-MDT0000 [ 8877.023553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.029796] Lustre: Skipped 64 previous similar messages [ 8877.050696] Lustre: server umount lustre-MDT0000 complete [ 8889.866827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8889.989815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8889.991768] Lustre: Skipped 21 previous similar messages [ 8890.025426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8890.027937] Lustre: Skipped 21 previous similar messages [ 8891.466939] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8892.573086] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8892.576488] Lustre: Skipped 21 previous similar messages [ 8895.457720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8895.460045] Lustre: Skipped 68 previous similar messages [ 8895.474062] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8895.476715] Lustre: Skipped 21 previous similar messages [ 8895.491757] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13428 to 0x2c0000401:13569) [ 8900.853176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8902.811150] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8903.548686] Lustre: lustre-OST0000: Denying connection for new client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 8908.957285] Lustre: lustre-OST0000: Denying connection for new client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 8919.197308] Lustre: lustre-OST0000: Denying connection for new client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 8919.201908] Lustre: Skipped 1 previous similar message [ 8939.676961] Lustre: lustre-OST0000: Denying connection for new client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 8939.681837] Lustre: Skipped 3 previous similar messages [ 8972.500158] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8972.502034] Lustre: 267861:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client d063271f-0a4b-48af-8510-377f4b22180c@ [ 8972.505080] Lustre: 267861:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8972.507123] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8972.514405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13620 to 0x280000401:13641) [ 8977.191131] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 8985.156367] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 8987.298578] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 02:37:59 (1783147079) [ 8988.697734] Lustre: Failing over lustre-OST0001 [ 8988.756237] Lustre: server umount lustre-OST0001 complete [ 9001.858508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9003.930362] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9007.994624] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 02:38:20 (1783147100) [ 9009.306836] Lustre: Failing over lustre-OST0000 [ 9009.351649] Lustre: server umount lustre-OST0000 complete [ 9022.128243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9023.713904] LustreError: 271051:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9023.716433] LustreError: 271051:0:(ldlm_lib.c:2891:target_recovery_thread()) Skipped 77 previous similar messages [ 9024.090806] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9030.111170] Lustre: *** cfs_fail_loc=715, val=40*** [ 9038.815918] Lustre: 3639:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147116/real 1783147116] req@ffff9dbd4ce58000 x1869755956163328/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783147132 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 9038.822566] Lustre: 3639:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 9038.824751] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:25 [ 9038.827579] Lustre: Skipped 1 previous similar message [ 9044.959173] Lustre: *** cfs_fail_loc=715, val=40*** [ 9044.960961] Lustre: Skipped 2 previous similar messages [ 9055.199568] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 9055.202204] Lustre: Skipped 2 previous similar messages [ 9061.343175] Lustre: *** cfs_fail_loc=715, val=40*** [ 9061.344506] Lustre: Skipped 2 previous similar messages [ 9063.759179] LustreError: 271051:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9063.763016] LustreError: 271051:0:(ldlm_lib.c:2891:target_recovery_thread()) Skipped 76 previous similar messages [ 9065.826935] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9066.352724] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9069.729684] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 02:39:22 (1783147162) [ 9071.153660] Lustre: Failing over lustre-MDT0000 [ 9071.459648] Lustre: server umount lustre-MDT0000 complete [ 9084.277695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.313971] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9084.316690] LustreError: Skipped 10 previous similar messages [ 9085.815573] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9089.508932] LustreError: 272599:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9092.575132] Lustre: *** cfs_fail_loc=715, val=80*** [ 9092.576595] Lustre: Skipped 2 previous similar messages [ 9102.493182] Lustre: lustre-MDT0000: Client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 1:28 [ 9102.496406] Lustre: Skipped 1 previous similar message [ 9104.863777] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9108.959194] Lustre: *** cfs_fail_loc=715, val=80*** [ 9108.961619] Lustre: Skipped 1 previous similar message [ 9117.852811] Lustre: lustre-MDT0000: Client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [ 9121.247692] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9124.319181] Lustre: *** cfs_fail_loc=715, val=80*** [ 9124.322049] Lustre: Skipped 1 previous similar message [ 9134.237045] Lustre: lustre-MDT0000: Client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 9137.631546] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9140.703127] Lustre: *** cfs_fail_loc=715, val=80*** [ 9140.704626] Lustre: Skipped 1 previous similar message [ 9150.620948] Lustre: lustre-MDT0000: Client d51beda4-3fdb-4ae1-9554-18a8d2cfc1ca (at 192.168.202.53@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 9152.991527] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9169.375622] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9169.591189] LustreError: 272599:0:(ldlm_lib.c:2891:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9169.597610] Lustre: 272599:0:(ldlm_lib.c:2937:target_recovery_thread()) too long recovery - read logs [ 9169.600943] LustreError: dumping log to /tmp/lustre-log.1783147262.272599 [ 9169.669591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13582 to 0x2c0000401:13601) [ 9169.669611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13655 to 0x280000401:13673) [ 9171.659481] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9172.237033] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9175.831447] Lustre: DEBUG MARKER: == replay-single test complete, duration 8866 sec ======== 02:41:08 (1783147268) [ 9176.447417] Lustre: DEBUG MARKER: === replay-single: start cleanup 02:41:09 (1783147269) === [ 9179.173655] Lustre: DEBUG MARKER: === replay-single: finish cleanup 02:41:11 (1783147271) === [ 9180.010294] Lustre: Failing over lustre-MDT0000 [ 9180.127512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9180.130521] LustreError: Skipped 14 previous similar messages [ 9180.131867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9180.133906] Lustre: Skipped 12 previous similar messages [ 9185.786309] Lustre: server umount lustre-MDT0000 complete [ 9202.550643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.306945] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9207.805826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13655 to 0x280000401:13705) [ 9207.805827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13582 to 0x2c0000401:13633) [ 9210.049761] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9210.662608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9213.516095] Lustre: server umount lustre-MDT0000 complete [ 9216.732076] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147309 with bad export cookie 16226753928955104563 [ 9216.741045] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9216.846738] Lustre: server umount lustre-MDT0001 complete [ 9240.319831] Lustre: server umount lustre-OST0000 complete [ 9243.025824] Lustre: server umount lustre-OST0001 complete [ 9248.674802] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 9249.763923] Key type lgssc unregistered [ 9249.894504] LNet: 277406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9249.897296] LNetError: 277406:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9249.904540] LNet: Removed LNI 192.168.202.153@tcp [ 9250.312116] Key type .llcrypt unregistered [ 9250.313161] Key type ._llcrypt unregistered