[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493130592 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003297] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007894] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013080] *** VALIDATE selinux *** [ 0.021802] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026207] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.030464] *** VALIDATE proc *** [ 0.031248] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035006] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038014] Speculative Store Bypass: Vulnerable [ 0.040991] debug: unmapping init [mem 0xffffffffa1859000-0xffffffffa1860fff] [ 0.042231] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043610] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051318] rcu: Hierarchical SRCU implementation. [ 0.053573] smp: Bringing up secondary CPUs ... [ 0.054617] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.059177] smp: Brought up 1 node, 4 CPUs [ 0.061019] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.211156] node 0 deferred pages initialised in 145ms [ 0.215013] devtmpfs: initialized [ 0.216254] x86/mm: Memory block size: 128MB [ 0.219026] gcov: version magic: 0x41383552 [ 0.221391] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.224142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.227454] pinctrl core: initialized pinctrl subsystem [ 0.229233] [ 0.229666] ************************************************************* [ 0.232021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.234016] ** ** [ 0.237021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.239015] ** ** [ 0.240012] ** This means that this kernel is built to expose internal ** [ 0.243016] ** IOMMU data structures, which may compromise security on ** [ 0.244014] ** your system. ** [ 0.246017] ** ** [ 0.248013] ** If you see this message and you are not debugging the ** [ 0.250017] ** kernel, report this immediately to your vendor! ** [ 0.252018] ** ** [ 0.254018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.256020] ************************************************************* [ 0.259019] NET: Registered protocol family 16 [ 0.260523] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.265076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.267112] cpuidle: using governor menu [ 0.268517] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269754] PCI: Using configuration type 1 for base access [ 0.270138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.279062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285149] cryptd: max_cpu_qlen set to 1000 [ 0.288290] ACPI: Added _OSI(Module Device) [ 0.290017] ACPI: Added _OSI(Processor Device) [ 0.291098] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.292012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296091] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.301417] ACPI: Interpreter enabled [ 0.303064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.303976] ACPI: Using IOAPIC for interrupt routing [ 0.305116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.307520] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.316820] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.319135] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.323098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.326431] acpiphp: Slot [2] registered [ 0.328113] acpiphp: Slot [5] registered [ 0.329206] acpiphp: Slot [6] registered [ 0.330083] acpiphp: Slot [7] registered [ 0.331076] acpiphp: Slot [8] registered [ 0.332061] acpiphp: Slot [9] registered [ 0.333053] acpiphp: Slot [10] registered [ 0.334087] acpiphp: Slot [3] registered [ 0.335083] acpiphp: Slot [4] registered [ 0.336110] acpiphp: Slot [11] registered [ 0.337080] acpiphp: Slot [12] registered [ 0.338078] acpiphp: Slot [13] registered [ 0.339087] acpiphp: Slot [14] registered [ 0.340080] acpiphp: Slot [15] registered [ 0.341095] acpiphp: Slot [16] registered [ 0.342213] acpiphp: Slot [17] registered [ 0.343086] acpiphp: Slot [18] registered [ 0.344079] acpiphp: Slot [19] registered [ 0.345084] acpiphp: Slot [20] registered [ 0.346049] acpiphp: Slot [21] registered [ 0.347078] acpiphp: Slot [22] registered [ 0.348126] acpiphp: Slot [23] registered [ 0.349118] acpiphp: Slot [24] registered [ 0.350207] acpiphp: Slot [25] registered [ 0.351119] acpiphp: Slot [26] registered [ 0.353081] acpiphp: Slot [27] registered [ 0.354080] acpiphp: Slot [28] registered [ 0.355080] acpiphp: Slot [29] registered [ 0.356112] acpiphp: Slot [30] registered [ 0.357098] acpiphp: Slot [31] registered [ 0.358067] PCI host bridge to bus 0000:00 [ 0.359021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.361022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.362019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.363024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.365032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368238] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.370021] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.373409] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.383014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396551] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.397481] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401692] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.405035] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.413020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.417019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.421637] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465200] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.473028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.499339] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.524151] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.529023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.533021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.544023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.551031] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.557025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.563024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.591517] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.601023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.613021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.622287] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.624335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.626314] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.628291] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.630221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.634250] iommu: Default domain type: Passthrough [ 0.635428] SCSI subsystem initialized [ 0.637158] ACPI: bus type USB registered [ 0.638056] usbcore: registered new interface driver usbfs [ 0.639072] usbcore: registered new interface driver hub [ 0.640071] usbcore: registered new device driver usb [ 0.641166] pps_core: LinuxPPS API ver. 1 registered [ 0.643012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.644175] PTP clock support registered [ 0.646161] EDAC MC: Ver: 3.0.0 [ 0.648105] PCI: Using ACPI for IRQ routing [ 0.649823] NetLabel: Initializing [ 0.651015] NetLabel: domain hash size = 128 [ 0.652018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.654127] NetLabel: unlabeled traffic allowed by default [ 0.657111] vgaarb: loaded [ 0.658290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.660015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.669257] clocksource: Switched to clocksource kvm-clock [ 0.774561] VFS: Disk quotas dquot_6.6.0 [ 0.776302] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.778705] *** VALIDATE ramfs *** [ 0.779952] *** VALIDATE hugetlbfs *** [ 0.781629] pnp: PnP ACPI init [ 0.784046] pnp: PnP ACPI: found 6 devices [ 0.802044] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.805797] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.807849] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.810121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.812052] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.814044] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.816797] NET: Registered protocol family 2 [ 0.819115] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.823749] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.827454] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.832810] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.835546] TCP: Hash tables configured (established 65536 bind 65536) [ 0.838538] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.841523] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.844559] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.847683] NET: Registered protocol family 1 [ 0.850381] RPC: Registered named UNIX socket transport module. [ 0.852576] RPC: Registered udp transport module. [ 0.854229] RPC: Registered tcp transport module. [ 0.855653] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.857328] NET: Registered protocol family 44 [ 0.858846] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.860921] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.862956] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.865232] PCI: CLS 0 bytes, default 64 [ 0.867608] Unpacking initramfs... [ 2.287365] debug: unmapping init [mem 0xffff95d2fcc54000-0xffff95d2fffbffff] [ 2.291247] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.293420] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.296468] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.801526] Initialise system trusted keyrings [ 2.803242] Key type blacklist registered [ 2.805284] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.816744] zbud: loaded [ 2.820288] *** VALIDATE nfs *** [ 2.821610] *** VALIDATE nfs4 *** [ 2.822874] pstore: using deflate compression [ 2.826142] Platform Keyring initialized [ 2.931708] NET: Registered protocol family 38 [ 2.933504] Key type asymmetric registered [ 2.935977] Asymmetric key parser 'x509' registered [ 2.938197] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.942288] io scheduler mq-deadline registered [ 2.943539] io scheduler kyber registered [ 2.944975] io scheduler bfq registered [ 2.947136] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.949829] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.952214] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.954299] ACPI: Power Button [PWRF] [ 2.959547] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.966389] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.978750] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.985128] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.999475] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.025038] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.053456] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.057375] Non-volatile memory driver v1.3 [ 3.058813] Linux agpgart interface v0.103 [ 3.096374] virtio_blk virtio1: [vda] 139368 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.099104] vda: detected capacity change from 0 to 71356416 [ 3.112607] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.115327] vdb: detected capacity change from 0 to 1073741824 [ 3.129212] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.131919] vdc: detected capacity change from 0 to 2621440000 [ 3.150903] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.153687] vdd: detected capacity change from 0 to 2621440000 [ 3.166815] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.169560] vde: detected capacity change from 0 to 4294967296 [ 3.187253] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.190763] vdf: detected capacity change from 0 to 4294967296 [ 3.200900] libphy: Fixed MDIO Bus: probed [ 3.208972] usbcore: registered new interface driver usbserial_generic [ 3.211567] usbserial: USB Serial support registered for generic [ 3.213912] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.218467] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.220668] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.225139] mousedev: PS/2 mouse device common for all mice [ 3.228433] rtc_cmos 00:05: RTC can wake from S4 [ 3.230185] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.232086] rtc_cmos 00:05: registered as rtc0 [ 3.235569] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.238283] intel_pstate: CPU model not supported [ 3.238789] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.243893] hid: raw HID events driver (C) Jiri Kosina [ 3.245727] usbcore: registered new interface driver usbhid [ 3.246170] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.247350] usbhid: USB HID core driver [ 3.251629] drop_monitor: Initializing network drop monitor service [ 3.253190] Initializing XFRM netlink socket [ 3.254558] NET: Registered protocol family 10 [ 3.256730] Segment Routing with IPv6 [ 3.257811] NET: Registered protocol family 17 [ 3.259353] mpls_gso: MPLS GSO support [ 3.264159] RAS: Correctable Errors collector initialized. [ 3.265638] AVX version of gcm_enc/dec engaged. [ 3.266899] AES CTR mode by8 optimization enabled [ 3.342565] sched_clock: Marking stable (3342531469, 0)->(4291764664, -949233195) [ 3.345440] registered taskstats version 1 [ 3.347496] Loading compiled-in X.509 certificates [ 3.349156] zswap: loaded using pool lzo/zbud [ 3.373627] Key type big_key registered [ 3.387752] Key type encrypted registered [ 3.389504] ima: No TPM chip found, activating TPM-bypass! [ 3.391428] ima: Allocated hash algorithm: sha1 [ 3.394129] ima: No architecture policies found [ 3.395871] evm: Initialising EVM extended attributes: [ 3.397761] evm: security.selinux [ 3.398805] evm: security.ima [ 3.399977] evm: security.capability [ 3.401602] evm: HMAC attrs: 0x1 [ 3.404110] rtc_cmos 00:05: setting system clock to 2026-02-10 00:32:25 UTC (1770683545) [ 3.410149] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 3.413316] debug: unmapping init [mem 0xffffffffa1582000-0xffffffffa1858fff] [ 3.420217] Write protecting the kernel read-only data: 28672k [ 3.423438] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 3.426359] debug: unmapping init [mem 0xffffffffa0514000-0xffffffffa05fffff] [ 3.476053] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.484045] systemd[1]: Detected virtualization kvm. [ 3.486267] systemd[1]: Detected architecture x86-64. [ 3.488173] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.512607] systemd[1]: No hostname configured. [ 3.514418] systemd[1]: Set hostname to . [ 3.516364] random: systemd: uninitialized urandom read (16 bytes read) [ 3.518640] systemd[1]: Initializing machine ID from random generator. [ 3.576718] random: ln: uninitialized urandom read (6 bytes read) [ 3.678428] random: systemd: uninitialized urandom read (16 bytes read) [ 3.680734] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.684514] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.688855] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.330670] device-mapper: uevent: version 1.0.3 [ 4.332965] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.108691] virtio_net virtio0 ens2: renamed from eth0 [ 5.127454] random: fast init done [ 5.185931] scsi host0: ata_piix [ 5.195379] scsi host1: ata_piix [ 5.196977] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.199352] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.836506] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.911262] random: crng init done [ 9.912600] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.315846] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.394319] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.615423] SELinux: Disabled at runtime. [ 11.675379] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.683513] systemd[1]: Detected virtualization kvm. [ 11.684799] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.147181] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.150682] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.155847] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.160195] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.163145] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.169734] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.176351] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.270514] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.550098] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.843606] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.876645] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.962917] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.987677] EDAC sbridge: Ver: 1.1.2 [ 14.538916] Key type dns_resolver registered [ 14.872620] NFS: Registering the id_resolver key type [ 14.874675] Key type id_resolver registered [ 14.876312] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg260-server login: [ 40.255913] libcfs: loading out-of-tree module taints kernel. [ 40.278472] Key type ._llcrypt registered [ 40.279678] Key type .llcrypt registered [ 40.327992] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 48.862643] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 49.598093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.606545] alg: No test for adler32 (adler32-zlib) [ 50.658907] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 51.001826] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 52.632220] Key type lgssc registered [ 55.237485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.346128] hrtimer: interrupt took 2112822 ns [ 77.743264] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.056726] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 147.318877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.350849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.779941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.838735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.946946] Lustre: lustre-MDT0000: new disk, initializing [ 149.087834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.116122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.873664] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.421302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 170.514691] Lustre: 6478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.544360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.548507] Lustre: Skipped 1 previous similar message [ 170.630945] Lustre: lustre-MDT0001: new disk, initializing [ 170.723055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.754096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.772426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.564252] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.370119] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.707384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 193.116104] Lustre: lustre-OST0000: new disk, initializing [ 193.121104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.135058] Lustre: 8381:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.235700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.705540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.712603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.872801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.229143] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.600400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 217.729234] Lustre: lustre-OST0001: new disk, initializing [ 217.732859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 217.742716] Lustre: 9436:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.813932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.134799] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.444372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 227.449876] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 227.530553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.894187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.453749] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.450130] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.779648] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing yml_node [ 271.142045] Lustre: DEBUG MARKER: Client: 2.17.50.65 [ 273.926596] Lustre: DEBUG MARKER: MDS: 2.17.50.65 [ 276.612311] Lustre: DEBUG MARKER: OSS: 2.17.50.65 [ 278.290610] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Mon Feb 9 19:36:59 EST 2026 [ 297.995414] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 300.046030] Lustre: DEBUG MARKER: === replay-single: start setup 19:37:20 (1770683840) === [ 304.455817] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing check_config_client /mnt/lustre [ 326.254388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 331.287578] Lustre: 13147:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 336.654615] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 345.714245] Lustre: DEBUG MARKER: === replay-single: finish setup 19:38:05 (1770683885) === [ 347.497546] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 19:38:08 (1770683888) [ 356.509366] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 358.779812] Lustre: Failing over lustre-MDT0000 [ 358.879237] LustreError: 14022:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 358.980237] Lustre: server umount lustre-MDT0000 complete [ 360.418708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 360.427124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.929747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.774922] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.806901] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 370.901397] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.923931] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 376.023369] LustreError: 6485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.042991] LustreError: 6485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 377.317250] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770683903/real 1770683903] req@ffff95d248f17b80 x1856696319335552/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770683919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 377.359613] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.133640] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.157594] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 385.637318] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 385.644780] LDISKFS-fs (dm-0): recovery complete [ 385.655606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.275371] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.314603] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 387.783961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.692683] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 392.786724] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 393.201249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 393.271721] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 401.839803] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 404.312954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 413.923901] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 19:39:14 (1770683954) [ 416.277740] Lustre: Failing over lustre-OST0000 [ 416.332225] LustreError: 15454:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 416.334988] LustreError: 15454:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 416.413191] Lustre: server umount lustre-OST0000 complete [ 418.792241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.813035] Lustre: Skipped 2 previous similar messages [ 418.826113] LustreError: 8366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.857975] LustreError: 8366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 435.402624] LustreError: 8364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.415670] LustreError: 8364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 437.936063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 438.334915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.972359] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 440.272231] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 440.274243] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 440.302924] Lustre: Skipped 3 previous similar messages [ 446.972624] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 457.125924] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 460.145928] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 472.570737] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 19:40:12 (1770684012) [ 483.379117] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 486.192890] Lustre: Failing over lustre-MDT0000 [ 486.359717] LustreError: 17161:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 486.370427] LustreError: 17161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 486.537798] Lustre: server umount lustre-MDT0000 complete [ 489.452539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.458959] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.470548] Lustre: Skipped 3 previous similar messages [ 489.502399] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 505.827642] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684031/real 1770684031] req@ffff95d2479ea680 x1856696319406080/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 505.842314] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.504268] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 511.506850] LDISKFS-fs (dm-0): recovery complete [ 511.516580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.362877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.680618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 520.682274] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 520.685718] Lustre: Skipped 1 previous similar message [ 520.806800] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 524.101932] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 529.108183] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 534.227578] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 539.350051] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 544.459278] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 554.699238] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:35 [ 554.721936] Lustre: Skipped 1 previous similar message [ 575.225777] Lustre: lustre-MDT0000: Denying connection for new client 8128459b-99db-4a71-8f38-a1aaf9a07c68 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:15 [ 575.259528] Lustre: Skipped 3 previous similar messages [ 590.503595] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 590.509327] Lustre: 17812:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9f8b4d35-7bd7-4676-9fcc-a9690ab0b466@ [ 590.527077] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 590.554690] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 590.556288] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 590.566756] Lustre: Skipped 2 previous similar messages [ 590.612619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 590.617300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 607.919759] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 19:42:28 (1770684148) [ 616.933453] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 619.779940] Lustre: Failing over lustre-MDT0000 [ 619.985485] LustreError: 18869:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 619.993865] LustreError: 18869:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 620.120803] Lustre: server umount lustre-MDT0000 complete [ 621.538206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.549735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.559184] Lustre: Skipped 1 previous similar message [ 621.569813] LustreError: 16433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.593268] LustreError: 16433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 639.456179] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684165/real 1770684165] req@ffff95d24792c700 x1856696319469568/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 639.483278] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.165688] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 644.169102] LDISKFS-fs (dm-0): recovery complete [ 644.174923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.065269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.114649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 655.169512] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 655.329437] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 655.337424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 658.303832] Lustre: lustre-MDT0000: Denying connection for new client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 658.332253] Lustre: Skipped 3 previous similar messages [ 725.198609] Lustre: lustre-MDT0000: Denying connection for new client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 725.222858] Lustre: Skipped 12 previous similar messages [ 725.501248] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 725.504427] Lustre: 19521:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8128459b-99db-4a71-8f38-a1aaf9a07c68@ [ 725.512804] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 725.538173] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 725.539498] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 725.555028] Lustre: Skipped 2 previous similar messages [ 725.599657] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 725.605651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 743.082561] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 19:44:43 (1770684283) [ 751.841257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 754.492618] Lustre: Failing over lustre-MDT0000 [ 754.630713] LustreError: 20585:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 754.640451] LustreError: 20585:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 754.948840] Lustre: server umount lustre-MDT0000 complete [ 755.958404] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.978732] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 756.227681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.241157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.268327] Lustre: Skipped 3 previous similar messages [ 773.096823] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684299/real 1770684299] req@ffff95d37daf2d80 x1856696319531264/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 773.133652] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 780.936058] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 780.938376] LDISKFS-fs (dm-0): recovery complete [ 780.946343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.072268] LustreError: 21203:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 783.084496] LustreError: 21203:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff95d37fb3d880 x1856696319536640/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1770684325 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 783.112117] LustreError: 21203:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 783.622186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.653537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 784.055455] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 788.646413] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 788.984437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 789.085365] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 789.180579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 789.188861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 798.503877] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 800.832515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 810.774168] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 19:45:51 (1770684351) [ 819.289802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 821.300866] Lustre: Failing over lustre-MDT0000 [ 821.477903] LustreError: 22406:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 821.487637] LustreError: 22406:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 821.683632] Lustre: server umount lustre-MDT0000 complete [ 824.807316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.817191] Lustre: Skipped 6 previous similar messages [ 840.172260] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684366/real 1770684366] req@ffff95d37d933100 x1856696319566464/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 840.216255] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 845.831396] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 845.834708] LDISKFS-fs (dm-0): recovery complete [ 845.863652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.400772] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d376182680 x1856696319573760/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 850.877584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 851.582379] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 856.056421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 856.077768] Lustre: Skipped 3 previous similar messages [ 856.190524] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 856.249122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 856.249870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 856.250601] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 865.730451] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 868.062047] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 878.703271] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 19:46:59 (1770684419) [ 887.381596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 889.427973] Lustre: Failing over lustre-MDT0000 [ 889.588640] LustreError: 24234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 889.598533] LustreError: 24234:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 889.837644] Lustre: server umount lustre-MDT0000 complete [ 891.874804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.900878] Lustre: Skipped 2 previous similar messages [ 908.259320] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684433/real 1770684433] req@ffff95d375868000 x1856696319602560/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 908.308692] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.839740] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 913.843480] LDISKFS-fs (dm-0): recovery complete [ 913.858746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 918.506144] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d248e62d80 x1856696319612160/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 918.813761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.821959] Lustre: Skipped 1 previous similar message [ 918.872436] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 920.385645] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 923.992823] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 924.156388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 924.159249] Lustre: Skipped 3 previous similar messages [ 924.308545] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 924.405428] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 924.405789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 933.703167] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 936.012518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 945.579937] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 19:48:06 (1770684486) [ 954.825668] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 956.883744] Lustre: Failing over lustre-MDT0000 [ 957.024549] LustreError: 26063:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 957.053680] LustreError: 26063:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 957.182492] Lustre: server umount lustre-MDT0000 complete [ 959.969522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.980971] Lustre: Skipped 2 previous similar messages [ 976.353835] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684502/real 1770684502] req@ffff95d37da8a680 x1856696319640064/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684518 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 976.372848] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 979.796417] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 979.799400] LDISKFS-fs (dm-0): recovery complete [ 979.806200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.922897] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 988.379994] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 992.149945] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 992.375267] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 992.448888] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 992.450613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1000.942081] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1003.528561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1013.650865] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 19:49:14 (1770684554) [ 1021.392125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1023.341058] Lustre: Failing over lustre-MDT0000 [ 1023.458224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1023.469591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.520393] LustreError: 27889:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1023.526919] LustreError: 27889:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1023.611861] Lustre: server umount lustre-MDT0000 complete [ 1024.240582] LustreError: 6485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.263150] LustreError: 6485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 130 previous similar messages [ 1044.448689] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684570/real 1770684570] req@ffff95d37df44e00 x1856696319678720/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1044.469878] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.597082] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1049.598878] LDISKFS-fs (dm-0): recovery complete [ 1049.606230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.706072] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f77242e [ 1054.722939] Lustre: MGC192.168.202.160@tcp: Connection restored to 0@lo (at 0@lo) [ 1054.731129] Lustre: Skipped 7 previous similar messages [ 1055.153919] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1056.526733] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1060.241521] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1060.451728] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1060.510364] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1060.520532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1068.602917] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.236900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1080.277994] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 19:50:20 (1770684620) [ 1081.347666] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1081.359094] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1081.370681] LustreError: 25971:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d248d2f800 x1856696309935104/t38654705666(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:499/0 lens 392/456 e 0 to 0 dl 1770684634 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1091.061364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1093.692987] Lustre: Failing over lustre-MDT0000 [ 1094.039522] Lustre: server umount lustre-MDT0000 complete [ 1096.161671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.185859] Lustre: Skipped 6 previous similar messages [ 1112.545550] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684638/real 1770684638] req@ffff95d37f10dc00 x1856696319718784/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1112.580339] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1119.119644] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1119.123111] LDISKFS-fs (dm-0): recovery complete [ 1119.168477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.787613] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d35fd0e300 x1856696319728256/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1123.288529] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1127.487484] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1128.565182] Lustre: 25971:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d3802f1880 x1856696309935104/t38654705666(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:546/0 lens 392/456 e 0 to 0 dl 1770684681 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1128.610444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1128.610535] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1136.882841] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1139.229165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1148.365647] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 19:51:29 (1770684689) [ 1155.929646] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1158.115797] Lustre: Failing over lustre-MDT0000 [ 1158.237478] LustreError: 31597:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1158.243125] LustreError: 31597:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1160.362424] Lustre: server umount lustre-MDT0000 complete [ 1184.051278] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1184.055439] LDISKFS-fs (dm-0): recovery complete [ 1184.065871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.251614] LustreError: 32216:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1185.260680] LustreError: 32216:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff95d3489caa00 x1856696319760896/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1770684727 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1185.284173] LustreError: 32216:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1185.802377] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f772db9 [ 1186.162692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.170203] Lustre: Skipped 3 previous similar messages [ 1186.209674] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1186.442353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1186.451866] Lustre: Skipped 1 previous similar message [ 1191.518391] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1191.541482] Lustre: Skipped 1 previous similar message [ 1191.584249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1191.585771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1192.624929] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1202.317439] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1204.238518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1214.554134] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 19:52:35 (1770684755) [ 1224.292626] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1225.525037] Lustre: *** cfs_fail_loc=114, val=0*** [ 1229.363106] Lustre: Failing over lustre-MDT0000 [ 1229.827454] Lustre: server umount lustre-MDT0000 complete [ 1248.737630] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770684774/real 1770684774] req@ffff95d37dad9f80 x1856696319795328/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770684790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1248.748807] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1248.751490] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.754968] LustreError: Skipped 1 previous similar message [ 1258.030143] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1258.033659] LDISKFS-fs (dm-0): recovery complete [ 1258.049927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.748675] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1277.896084] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1279.166937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1279.169127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1286.494046] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1288.578949] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1300.940901] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 19:54:01 (1770684841) [ 1312.574187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1313.811307] Lustre: *** cfs_fail_loc=128, val=0*** [ 1316.582175] Lustre: Failing over lustre-MDT0000 [ 1316.818638] Lustre: server umount lustre-MDT0000 complete [ 1342.768883] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1342.772179] LDISKFS-fs (dm-0): recovery complete [ 1342.782186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.540576] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d377627800 x1856696319848960/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1346.776826] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (not set up) [ 1352.185251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1352.199497] Lustre: Skipped 17 previous similar messages [ 1352.349698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1352.357380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1352.436637] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1362.397327] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1364.530774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1376.118738] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 19:55:16 (1770684916) [ 1385.900392] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1388.805468] Lustre: Failing over lustre-MDT0000 [ 1389.178854] Lustre: server umount lustre-MDT0000 complete [ 1393.121610] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.125951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1393.134539] Lustre: Skipped 17 previous similar messages [ 1413.853152] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1413.856343] LDISKFS-fs (dm-0): recovery complete [ 1413.863124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.165696] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1420.171263] Lustre: Skipped 1 previous similar message [ 1425.521969] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1425.747176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1425.747488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1434.321218] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1436.178096] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1447.850598] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 19:56:27 (1770684987) [ 1457.911943] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1460.744090] Lustre: Failing over lustre-MDT0000 [ 1460.966886] LustreError: 39094:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1460.970771] LustreError: 39094:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 1461.087627] Lustre: server umount lustre-MDT0000 complete [ 1487.688557] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1487.691402] LDISKFS-fs (dm-0): recovery complete [ 1487.721177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.438773] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1503.455824] Lustre: Skipped 3 previous similar messages [ 1507.993356] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1507.998951] Lustre: Skipped 3 previous similar messages [ 1508.045654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1508.046101] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1508.529196] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1517.853096] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1520.061162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1531.927900] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 19:57:52 (1770685072) [ 1540.974426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1550.089770] Lustre: Failing over lustre-MDT0000 [ 1550.407264] Lustre: server umount lustre-MDT0000 complete [ 1553.902086] LustreError: 7670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.915190] LustreError: 7670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 272 previous similar messages [ 1569.248185] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770685095/real 1770685095] req@ffff95d2479bbb80 x1856696319965056/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770685111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1569.283298] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1569.290930] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.306898] LustreError: Skipped 3 previous similar messages [ 1573.508814] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1573.517962] LDISKFS-fs (dm-0): recovery complete [ 1573.541056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.488591] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d375b24000 x1856696319973120/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1585.011596] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1590.330199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1590.331233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1594.255324] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.247485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1622.498724] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 19:59:23 (1770685163) [ 1631.209435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1633.639510] Lustre: Failing over lustre-MDT0000 [ 1633.854192] Lustre: server umount lustre-MDT0000 complete [ 1659.676573] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1659.680107] LDISKFS-fs (dm-0): recovery complete [ 1659.689087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.181215] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (not set up) [ 1667.697360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1667.706684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1667.809966] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1676.919234] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1679.256223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1692.115952] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 20:00:32 (1770685232) [ 1700.655416] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1703.063395] Lustre: Failing over lustre-MDT0000 [ 1703.215473] Lustre: server umount lustre-MDT0000 complete [ 1703.396649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1727.826781] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1727.828965] LDISKFS-fs (dm-0): recovery complete [ 1727.845584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.419087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.433522] Lustre: Skipped 6 previous similar messages [ 1734.509368] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1734.516986] Lustre: Skipped 3 previous similar messages [ 1739.944906] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1739.993597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1739.998290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1749.161285] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.081945] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1761.144973] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 20:01:41 (1770685301) [ 1769.991473] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1772.273584] Lustre: Failing over lustre-MDT0000 [ 1774.587349] Lustre: server umount lustre-MDT0000 complete [ 1799.284591] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1799.295318] LDISKFS-fs (dm-0): recovery complete [ 1799.305444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.455623] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (not set up) [ 1807.549538] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1807.980972] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1807.985797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1817.231461] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1819.573535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1830.425655] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 20:02:50 (1770685370) [ 1838.593204] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1841.267804] Lustre: Failing over lustre-MDT0000 [ 1841.741249] Lustre: server umount lustre-MDT0000 complete [ 1843.681746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.219553] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1867.226251] LDISKFS-fs (dm-0): recovery complete [ 1867.235444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.291816] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f77f4c2 [ 1869.317308] Lustre: MGC192.168.202.160@tcp: Connection restored to 0@lo (at 0@lo) [ 1869.323707] Lustre: Skipped 27 previous similar messages [ 1874.848485] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1875.111337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1875.116891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1884.311046] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1886.597661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1896.985393] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 20:03:57 (1770685437) [ 1905.336522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1907.855011] Lustre: Failing over lustre-MDT0000 [ 1908.300708] Lustre: server umount lustre-MDT0000 complete [ 1910.755357] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.780270] Lustre: Skipped 26 previous similar messages [ 1931.640420] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1931.642945] LDISKFS-fs (dm-0): recovery complete [ 1931.649958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.611747] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1942.222698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1942.225551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1951.215313] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1953.345867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1964.510795] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 20:05:04 (1770685504) [ 1975.575870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1978.815040] Lustre: Failing over lustre-MDT0000 [ 1978.969589] LustreError: 51941:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1978.972457] LustreError: 51941:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 1979.374904] Lustre: server umount lustre-MDT0000 complete [ 2003.791206] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2003.794829] LDISKFS-fs (dm-0): recovery complete [ 2003.812337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.823456] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2014.907242] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 2014.911309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 2022.484689] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.635266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2035.092190] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 20:06:15 (1770685575) [ 2043.475967] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2045.583951] Lustre: Failing over lustre-MDT0000 [ 2045.921925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2046.071323] Lustre: server umount lustre-MDT0000 complete [ 2069.409083] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2069.412309] LDISKFS-fs (dm-0): recovery complete [ 2069.419304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.432688] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2079.437521] Lustre: Skipped 7 previous similar messages [ 2082.345506] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2082.970642] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2082.978348] Lustre: Skipped 7 previous similar messages [ 2083.015784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 2083.027278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 2090.205891] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2091.872583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2100.663384] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 20:07:21 (1770685641) [ 2109.354975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2111.675502] Lustre: Failing over lustre-MDT0000 [ 2111.922653] Lustre: server umount lustre-MDT0000 complete [ 2129.365361] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770685655/real 1770685655] req@ffff95d3759dc380 x1856696320324352/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770685671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2129.399327] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2129.408103] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.423833] LustreError: Skipped 7 previous similar messages [ 2134.059973] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2134.064679] LDISKFS-fs (dm-0): recovery complete [ 2134.080890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.625047] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f780aef [ 2144.153836] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2145.356324] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2145.361130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2151.330893] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2153.092372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2162.960389] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 20:08:23 (1770685703) [ 2171.073412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2173.454261] Lustre: Failing over lustre-MDT0000 [ 2174.157340] Lustre: server umount lustre-MDT0000 complete [ 2175.968529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2175.977508] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.019370] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 297 previous similar messages [ 2198.627117] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2198.630242] LDISKFS-fs (dm-0): recovery complete [ 2198.638513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.598505] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d36fdd6a00 x1856696320368768/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2206.491614] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2208.288087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 2208.294821] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 2213.970742] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2215.726255] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2224.694799] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 20:09:25 (1770685765) [ 2231.695919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2233.588474] Lustre: Failing over lustre-MDT0000 [ 2233.828205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.837713] Lustre: Skipped 3 previous similar messages [ 2234.042955] Lustre: server umount lustre-MDT0000 complete [ 2257.073743] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2257.081718] LDISKFS-fs (dm-0): recovery complete [ 2257.097303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.066646] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f7815c3 [ 2265.293214] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (not set up) [ 2265.495130] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2265.506491] Lustre: Skipped 7 previous similar messages [ 2269.487934] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2270.800970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2270.808487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 2277.360607] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2279.168434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2286.995309] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 20:10:27 (1770685827) [ 2294.435730] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2296.200030] Lustre: Failing over lustre-MDT0000 [ 2296.288878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2296.296239] Lustre: Skipped 1 previous similar message [ 2296.426528] Lustre: server umount lustre-MDT0000 complete [ 2318.282414] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2318.287494] LDISKFS-fs (dm-0): recovery complete [ 2318.298439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.564945] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2332.820427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2332.822429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2340.512274] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2342.171195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2350.715645] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 20:11:31 (1770685891) [ 2357.682802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2359.523268] Lustre: Failing over lustre-MDT0000 [ 2359.798062] Lustre: server umount lustre-MDT0000 complete [ 2380.363692] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2380.365775] LDISKFS-fs (dm-0): recovery complete [ 2380.371436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.251770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.257435] Lustre: Skipped 9 previous similar messages [ 2394.778275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 2394.787114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 2395.113905] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2402.441214] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2404.337193] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2414.018712] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 20:12:34 (1770685954) [ 2421.777398] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2423.582262] Lustre: Failing over lustre-MDT0000 [ 2423.848700] Lustre: server umount lustre-MDT0000 complete [ 2425.321278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2425.329079] LustreError: Skipped 2 previous similar messages [ 2446.269721] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2446.272095] LDISKFS-fs (dm-0): recovery complete [ 2446.276940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.513620] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2456.697401] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2456.700382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 2464.248418] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2466.221745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2475.089900] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 20:13:35 (1770686015) [ 2482.079953] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2484.077550] Lustre: Failing over lustre-MDT0000 [ 2484.344188] Lustre: server umount lustre-MDT0000 complete [ 2507.379320] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2507.381664] LDISKFS-fs (dm-0): recovery complete [ 2507.387292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.380407] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f782b72 [ 2513.387753] Lustre: MGC192.168.202.160@tcp: Connection restored to 0@lo (at 0@lo) [ 2513.399381] Lustre: Skipped 42 previous similar messages [ 2517.602606] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2519.140394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2519.140797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2525.042067] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2526.750648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2534.356298] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 20:14:35 (1770686075) [ 2541.819503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2543.650928] Lustre: Failing over lustre-MDT0000 [ 2543.931495] Lustre: server umount lustre-MDT0000 complete [ 2544.616555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.626370] Lustre: Skipped 40 previous similar messages [ 2564.371668] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2564.383281] LDISKFS-fs (dm-0): recovery complete [ 2564.397000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.210622] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d37f027480 x1856696320575232/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2574.716823] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2575.987226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2575.989463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2581.764978] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2583.437906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2591.192290] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:15:32 (1770686132) [ 2597.229672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2599.067462] Lustre: Failing over lustre-MDT0000 [ 2599.234141] LustreError: 70209:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2599.238813] LustreError: 70209:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 2601.355873] Lustre: server umount lustre-MDT0000 complete [ 2622.436772] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2622.439581] LDISKFS-fs (dm-0): recovery complete [ 2622.454267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.678268] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2632.843917] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2632.844606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2638.231484] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2639.647671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2646.905681] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 20:16:27 (1770686187) [ 2649.956932] Lustre: 71684:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 34d78110-fa36-421a-b835-6ec18dfb5b21 at adminstrative request [ 2656.283820] Lustre: Failing over lustre-MDT0000 [ 2656.621146] Lustre: server umount lustre-MDT0000 complete [ 2674.722324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.388791] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f783ceb [ 2685.150317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2685.157357] Lustre: Skipped 9 previous similar messages [ 2689.027632] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2690.055373] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2690.064424] Lustre: Skipped 9 previous similar messages [ 2690.106553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2690.106870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2696.116965] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.499677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2702.568787] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2708.915170] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2714.954461] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 20:17:35 (1770686255) [ 2716.087931] Lustre: 73534:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 34d78110-fa36-421a-b835-6ec18dfb5b21 at adminstrative request [ 2725.293476] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:17:46 (1770686266) [ 2731.968611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2733.997252] Lustre: Failing over lustre-MDT0000 [ 2734.256401] Lustre: server umount lustre-MDT0000 complete [ 2736.100897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2736.108973] LustreError: Skipped 2 previous similar messages [ 2752.480260] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770686278/real 1770686278] req@ffff95d35fd3e300 x1856696320672128/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770686294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2752.499588] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2752.506939] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.513036] LustreError: Skipped 9 previous similar messages [ 2755.545857] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2755.548815] LDISKFS-fs (dm-0): recovery complete [ 2755.556410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.816369] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2768.503168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2768.503292] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2773.506340] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2774.910692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2782.904581] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:18:44 (1770686324) [ 2789.530721] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2791.578596] Lustre: Failing over lustre-MDT0000 [ 2792.019879] Lustre: server umount lustre-MDT0000 complete [ 2793.964228] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.974776] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 334 previous similar messages [ 2812.734658] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2812.737804] LDISKFS-fs (dm-0): recovery complete [ 2812.750539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.585381] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d36fd95f80 x1856696320712832/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2824.043620] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2826.316316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2826.330390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2830.908827] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2832.087376] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2839.636714] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:19:40 (1770686380) [ 2847.361280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2849.710040] Lustre: Failing over lustre-MDT0000 [ 2850.013030] Lustre: server umount lustre-MDT0000 complete [ 2871.790939] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2871.792814] LDISKFS-fs (dm-0): recovery complete [ 2871.799336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.524816] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f784d14 [ 2878.774629] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2878.782187] Lustre: Skipped 9 previous similar messages [ 2883.086753] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2884.215175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2884.215536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2890.382774] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2891.801721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2899.309044] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 20:20:40 (1770686440) [ 2905.897585] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2907.581284] Lustre: Failing over lustre-MDT0000 [ 2907.815396] Lustre: server umount lustre-MDT0000 complete [ 2928.627661] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2928.630733] LDISKFS-fs (dm-0): recovery complete [ 2928.640874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.299848] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d36fd96d80 x1856696320778112/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2940.176836] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2942.010177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2942.018758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2946.858572] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.215510] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2955.860860] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:21:36 (1770686496) [ 2962.808713] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2964.521770] Lustre: Failing over lustre-MDT0000 [ 2964.786968] Lustre: server umount lustre-MDT0000 complete [ 2985.788728] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2985.792392] LDISKFS-fs (dm-0): recovery complete [ 2985.802206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.429465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.432084] Lustre: Skipped 9 previous similar messages [ 2999.228609] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3000.882513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 3000.884219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 3005.215493] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3006.844969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3014.491144] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:22:35 (1770686555) [ 3022.530899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3024.510837] Lustre: Failing over lustre-MDT0000 [ 3024.725677] Lustre: server umount lustre-MDT0000 complete [ 3045.004838] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3045.010048] LDISKFS-fs (dm-0): recovery complete [ 3045.019949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.638742] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3057.742285] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 3057.749803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 3062.436965] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3063.881972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3070.972723] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:23:32 (1770686612) [ 3077.344590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3078.892562] Lustre: Failing over lustre-MDT0000 [ 3079.048307] Lustre: server umount lustre-MDT0000 complete [ 3099.799737] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3099.803061] LDISKFS-fs (dm-0): recovery complete [ 3099.820800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.859734] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d36fdd4e00 x1856696320875264/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3113.608786] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3115.494506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3115.509335] Lustre: Skipped 42 previous similar messages [ 3115.653280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3115.653493] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3120.280716] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3121.753453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3128.174329] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:24:29 (1770686669) [ 3134.642337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3136.622824] Lustre: Failing over lustre-MDT0000 [ 3136.722704] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 3137.047796] Lustre: server umount lustre-MDT0000 complete [ 3157.665733] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3157.668269] LDISKFS-fs (dm-0): recovery complete [ 3157.675448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.729234] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3173.492588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3173.493161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3179.859566] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3181.227783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3188.638759] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:25:29 (1770686729) [ 3195.549106] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3197.381696] Lustre: Failing over lustre-MDT0000 [ 3197.736517] Lustre: server umount lustre-MDT0000 complete [ 3198.945643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.958306] Lustre: Skipped 43 previous similar messages [ 3218.572247] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3218.576374] LDISKFS-fs (dm-0): recovery complete [ 3218.585652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.046301] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3232.304181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3232.307760] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3237.173360] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3238.465926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3245.175784] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:26:26 (1770686786) [ 3251.628797] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3253.206702] Lustre: Failing over lustre-MDT0000 [ 3253.323481] LustreError: 90638:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3253.328958] LustreError: 90638:0:(obd_class.h:479:obd_check_dev()) Skipped 87 previous similar messages [ 3253.420486] Lustre: server umount lustre-MDT0000 complete [ 3257.829664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3257.839289] LustreError: Skipped 7 previous similar messages [ 3273.739501] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3273.743190] LDISKFS-fs (dm-0): recovery complete [ 3273.752351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.283640] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3279.393400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3279.398918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3283.697622] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3284.907288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3291.974144] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:27:13 (1770686833) [ 3298.505648] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3300.258962] Lustre: Failing over lustre-MDT0000 [ 3300.560271] Lustre: server umount lustre-MDT0000 complete [ 3321.141823] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3321.145560] LDISKFS-fs (dm-0): recovery complete [ 3321.154247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.316957] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3325.641911] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3325.649363] Lustre: Skipped 10 previous similar messages [ 3327.004458] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3327.011786] Lustre: Skipped 10 previous similar messages [ 3327.057129] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3327.061459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3331.843724] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3333.396957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3340.947229] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:28:01 (1770686881) [ 3342.075638] Lustre: 93838:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 34d78110-fa36-421a-b835-6ec18dfb5b21 at adminstrative request [ 3351.045204] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:28:11 (1770686891) [ 3356.365885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3357.888856] Lustre: Failing over lustre-MDT0000 [ 3358.129736] Lustre: server umount lustre-MDT0000 complete [ 3369.025827] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3369.031373] LDISKFS-fs (dm-0): recovery complete [ 3369.046288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.215978] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3369.225171] LustreError: Skipped 10 previous similar messages [ 3369.419079] Lustre: lustre-MDT0000: Aborting client recovery [ 3369.423446] LustreError: 95120:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3369.426197] Lustre: 95152:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3369.434211] Lustre: 95152:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 34d78110-fa36-421a-b835-6ec18dfb5b21@ [ 3369.440199] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3369.445861] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3369.459917] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3369.512584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 3369.515984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1480 to 0x280000401:1505) [ 3372.657802] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3374.565354] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3391.446728] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 20:28:52 (1770686932) [ 3398.395846] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3400.252757] Lustre: Failing over lustre-MDT0000 [ 3400.493436] Lustre: server umount lustre-MDT0000 complete [ 3401.942913] LustreError: 7670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.952809] LustreError: 7670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 327 previous similar messages [ 3412.153954] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3412.156710] LDISKFS-fs (dm-0): recovery complete [ 3412.164651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.477210] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3412.508495] Lustre: lustre-MDT0000: Aborting client recovery [ 3412.511883] LustreError: 96967:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3412.517280] Lustre: 96999:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3412.522889] Lustre: 96999:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3412.528024] Lustre: 96999:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3412.537348] Lustre: 96999:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3412.543436] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3412.551515] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3412.565641] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3412.624849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3412.625464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3416.284140] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3417.578925] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3428.733795] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3428.736025] Lustre: Skipped 1 previous similar message [ 3434.175966] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 20:29:35 (1770686975) [ 3440.986870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3442.626634] Lustre: Failing over lustre-MDT0000 [ 3442.880122] Lustre: server umount lustre-MDT0000 complete [ 3453.526696] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3453.530848] LDISKFS-fs (dm-0): recovery complete [ 3453.539462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.785255] Lustre: lustre-MDT0000: Aborting client recovery [ 3453.787353] LustreError: 98816:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3453.790994] Lustre: 98848:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3453.793667] Lustre: 98848:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3453.805585] Lustre: 98848:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3453.810540] Lustre: 98848:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3453.817321] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3453.821872] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3453.831820] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3453.869862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 3453.877370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 3457.229470] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3459.048337] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3480.365184] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 20:30:21 (1770687021) [ 3481.217050] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3481.224202] LustreError: 13674:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d347a70700 x1856696311029632/t201863462916(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:634/0 lens 512/456 e 0 to 0 dl 1770687034 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3484.662218] Lustre: Failing over lustre-MDT0000 [ 3484.958457] Lustre: server umount lustre-MDT0000 complete [ 3492.040146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.419968] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3492.422790] Lustre: lustre-MDT0000: Aborting client recovery [ 3492.423621] Lustre: Skipped 17 previous similar messages [ 3492.434530] LustreError: 100122:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3492.439450] Lustre: 100154:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3492.444331] Lustre: 100154:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3492.448721] Lustre: 100154:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3492.455290] Lustre: 100154:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3492.459658] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3492.464589] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3492.475297] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3492.512900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 3492.514723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3496.186248] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3497.445352] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3515.404709] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3516.810757] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:30:57 (1770687057) [ 3523.405442] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3525.610641] Lustre: Failing over lustre-MDT0000 [ 3525.822743] Lustre: server umount lustre-MDT0000 complete [ 3536.162184] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3536.164995] LDISKFS-fs (dm-0): recovery complete [ 3536.174809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.590671] Lustre: lustre-MDT0000: Aborting client recovery [ 3536.593599] LustreError: 102068:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3536.600741] Lustre: 102100:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3536.611292] Lustre: 102100:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3536.620634] Lustre: 102100:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3536.631342] Lustre: 102100:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3536.638092] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3536.647215] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3536.661803] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3536.760995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3536.777765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 3540.212953] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3541.991416] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3557.595241] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:31:38 (1770687098) [ 3584.412437] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3585.738497] Lustre: Failing over lustre-MDT0000 [ 3585.977241] Lustre: server umount lustre-MDT0000 complete [ 3604.453086] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770687130/real 1770687130] req@ffff95d375b07100 x1856696321246208/t0(0) o400->MGC192.168.202.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1770687146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3604.478875] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 3605.208214] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3605.209964] LDISKFS-fs (dm-0): recovery complete [ 3605.215258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.690139] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d24dce8e00 x1856696321254784/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3614.952544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.957484] Lustre: Skipped 11 previous similar messages [ 3619.128559] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3620.435145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3620.438092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3625.241575] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3626.527752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3643.292623] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:33:04 (1770687184) [ 3662.101408] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3669.552982] Lustre: Failing over lustre-MDT0000 [ 3669.745651] Lustre: server umount lustre-MDT0000 complete [ 3688.622255] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3688.624900] LDISKFS-fs (dm-0): recovery complete [ 3688.631423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.147509] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcda98e4e0f7ae2e8 [ 3700.958609] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3706.043432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3706.056292] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3708.650636] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3709.921220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3727.457096] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:34:28 (1770687268) [ 3729.375429] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3730.265678] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3730.269655] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3730.283364] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3730.285914] Lustre: Skipped 48 previous similar messages [ 3735.302811] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 20:34:36 (1770687276) [ 3755.465549] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3763.885233] Lustre: Failing over lustre-OST0000 [ 3763.952187] Lustre: server umount lustre-OST0000 complete [ 3782.513458] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3782.514900] LDISKFS-fs (dm-2): recovery complete [ 3782.525557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3786.245298] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3839.922187] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 20:36:21 (1770687381) [ 3845.543315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3847.756904] Lustre: Failing over lustre-MDT0000 [ 3848.004761] Lustre: server umount lustre-MDT0000 complete [ 3849.185123] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3849.195112] Lustre: Skipped 42 previous similar messages [ 3866.629287] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3866.631265] LDISKFS-fs (dm-0): recovery complete [ 3866.636828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.297355] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d377798e00 x1856696321679872/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3878.221767] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3881.030072] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3881.030494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3883.892800] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3884.963785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3897.317332] LustreError: 111238:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3897.319894] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3898.339773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3900.564146] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 20:37:21 (1770687441) [ 3903.354503] LustreError: 13674:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3908.576116] LustreError: 13674:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3908.580674] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 3908.621087] LustreError: 40104:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 3909.906805] LustreError: 13674:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3915.232302] LustreError: 13674:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3915.239452] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 3916.497377] LustreError: 7670:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3921.889283] LustreError: 7670:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3921.897501] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 3923.274558] LustreError: 7670:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3928.544112] LustreError: 7670:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3929.689116] LustreError: 7670:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3935.200143] LustreError: 7670:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3935.204183] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 3935.208753] Lustre: Skipped 1 previous similar message [ 3942.994205] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3943.002455] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3948.512362] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3948.517240] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3955.169830] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 3955.177427] Lustre: Skipped 2 previous similar messages [ 3962.983279] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3962.986850] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3968.480184] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3968.489431] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3973.772322] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 20:38:35 (1770687515) [ 3974.884859] LustreError: 6485:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3985.100221] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 3990.221129] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 3995.338458] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 4000.459366] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 4000.462947] Lustre: Skipped 1 previous similar message [ 4005.579919] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 4014.984157] LustreError: 6485:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4014.989039] Lustre: 6485:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff95d380199c00 x1856696313680640/t0(0) o38->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:0/0 lens 520/416 e 0 to 0 dl 1770687536 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4015.823317] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 4015.830339] Lustre: Skipped 3 previous similar messages [ 4015.833219] LustreError: 7670:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4041.417982] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 4041.421604] Lustre: Skipped 1 previous similar message [ 4055.920263] LustreError: 7670:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4055.926499] Lustre: 7670:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff95d380231c00 x1856696313685120/t0(0) o38->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:0/0 lens 520/416 e 0 to 0 dl 1770687577 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4056.783869] LustreError: 7670:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4082.383031] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 4082.388953] Lustre: Skipped 2 previous similar messages [ 4096.872150] LustreError: 7670:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4096.878969] Lustre: 7670:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95d248f14000 x1856696313688192/t0(0) o38->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:0/0 lens 520/416 e 0 to 0 dl 1770687618 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4097.740678] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 4097.745546] Lustre: Skipped 1 previous similar message [ 4097.747786] LustreError: 13674:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4123.349658] Lustre: lustre-MDT0000: Export ffff95d24d96d800 already connecting from 192.168.202.60@tcp [ 4123.356079] Lustre: Skipped 3 previous similar messages [ 4137.752745] LustreError: 13674:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4137.759400] Lustre: 13674:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95d3802e4380 x1856696313691392/t0(0) o38->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:0/0 lens 520/416 e 0 to 0 dl 1770687659 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4138.699631] LustreError: 6484:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4178.763059] LustreError: 6484:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4178.770622] Lustre: 6484:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95d375babb80 x1856696313694592/t0(0) o38->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:0/0 lens 520/416 e 0 to 0 dl 1770687700 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4183.757896] LustreError: 114465:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4191.066827] LustreError: 114465:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 4193.713726] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 20:42:15 (1770687735) [ 4197.796290] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4200.326435] Lustre: Failing over lustre-MDT0000 [ 4200.403228] LustreError: 115324:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4200.407129] LustreError: 115324:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 4200.454864] Lustre: server umount lustre-MDT0000 complete [ 4202.465314] LustreError: 16433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.471821] LustreError: 16433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 153 previous similar messages [ 4207.142648] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4207.145853] LDISKFS-fs (dm-0): recovery complete [ 4207.156376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.223553] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4207.228209] LustreError: Skipped 7 previous similar messages [ 4207.293052] Lustre: *** cfs_fail_loc=712, val=0*** [ 4207.295674] LustreError: 42098:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff95d24e168380 x1856696321902720/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4207.360163] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4207.360658] Lustre: lustre-MDT0000: Aborting client recovery [ 4207.366215] Lustre: Skipped 9 previous similar messages [ 4207.367442] LustreError: 115971:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4207.368677] Lustre: 116003:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4207.373845] Lustre: 116003:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4207.376832] Lustre: 116003:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4207.382739] Lustre: 116003:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4207.386578] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4207.392756] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4207.404147] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4207.435669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4207.447705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4209.284904] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4212.706536] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4220.363530] Lustre: Failing over lustre-MDT0000 [ 4220.518966] Lustre: server umount lustre-MDT0000 complete [ 4222.947512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4222.951159] LustreError: Skipped 8 previous similar messages [ 4234.765123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.983349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.986961] Lustre: Skipped 4 previous similar messages [ 4236.189340] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4236.194103] Lustre: Skipped 4 previous similar messages [ 4236.875100] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4240.370096] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4240.374596] Lustre: Skipped 4 previous similar messages [ 4240.398303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4240.398312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4241.958041] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4242.794459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4247.593372] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 20:43:09 (1770687789) [ 4247.659918] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 4247.662770] Lustre: Skipped 2 previous similar messages [ 4251.622144] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 20:43:13 (1770687793) [ 4252.124076] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4252.126474] LustreError: 6493:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d36fd91f80 x1856696313752704/t0(0) o700->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:650/0 lens 264/248 e 0 to 0 dl 1770687805 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4269.370163] Lustre: Failing over lustre-MDT0000 [ 4269.531235] Lustre: server umount lustre-MDT0000 complete [ 4284.228990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.245650] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4289.539873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4289.542887] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4291.172161] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4291.993432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4297.176084] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:43:58 (1770687838) [ 4298.262289] Lustre: Failing over lustre-OST0000 [ 4298.329697] Lustre: server umount lustre-OST0000 complete [ 4311.919218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4314.414623] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4318.655122] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.365931] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4385.898778] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:45:27 (1770687927) [ 4389.118750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4389.999390] Lustre: Failing over lustre-MDT0000 [ 4390.158086] Lustre: server umount lustre-MDT0000 complete [ 4405.138733] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4405.141057] LDISKFS-fs (dm-0): recovery complete [ 4405.145900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.816246] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4410.342088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4410.345070] Lustre: Skipped 21 previous similar messages [ 4410.484272] Lustre: *** cfs_fail_loc=216, val=0*** [ 4410.484683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4410.486298] LustreError: 121774:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4411.553147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4473.801280] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 20:46:55 (1770688015) [ 4474.603283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.607262] Lustre: Skipped 19 previous similar messages [ 4482.421289] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 20:47:04 (1770688024) [ 4483.659576] Lustre: Failing over lustre-MDT0000 [ 4483.804586] Lustre: server umount lustre-MDT0000 complete [ 4497.271382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.376719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4497.380360] Lustre: Skipped 3 previous similar messages [ 4499.211819] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4502.505171] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4502.506896] LustreError: 123334:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d37f3f0380 x1856696313872896/t0(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:190/0 lens 328/344 e 0 to 0 dl 1770688100 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4563.145362] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 1:04 [ 4563.154053] Lustre: 123334:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4563.182585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4563.182626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4565.020247] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4565.750596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4571.113750] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 20:48:32 (1770688112) [ 4572.639053] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4576.122172] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4576.914918] Lustre: Failing over lustre-MDT0000 [ 4577.067708] Lustre: server umount lustre-MDT0000 complete [ 4592.060058] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4592.062179] LDISKFS-fs (dm-0): recovery complete [ 4592.066687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.790893] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4597.238570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4597.239271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 4598.761868] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4599.496536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4603.605359] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 20:49:05 (1770688145) [ 4604.089458] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4608.462573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4609.292551] Lustre: Failing over lustre-MDT0000 [ 4609.422854] Lustre: server umount lustre-MDT0000 complete [ 4624.702207] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4624.704256] LDISKFS-fs (dm-0): recovery complete [ 4624.708453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.540698] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4630.015655] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4630.015702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 4631.666472] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4632.472964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4637.093268] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 20:49:38 (1770688178) [ 4637.676201] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4642.884095] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4643.875973] Lustre: Failing over lustre-MDT0000 [ 4643.986222] Lustre: server umount lustre-MDT0000 complete [ 4659.325156] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4659.327059] LDISKFS-fs (dm-0): recovery complete [ 4659.333831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.257811] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4664.837359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3201) [ 4664.837366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4670.472126] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 20:50:12 (1770688212) [ 4671.928433] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4671.930204] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4671.932169] LustreError: 6486:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d37da3f100 x1856696313924224/t257698037777(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:315/0 lens 392/456 e 0 to 0 dl 1770688225 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4673.131463] Lustre: Failing over lustre-MDT0000 [ 4673.293093] Lustre: server umount lustre-MDT0000 complete [ 4686.860072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.586951] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4692.482928] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d377799180 x1856696313924224/t257698037777(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:335/0 lens 392/456 e 0 to 0 dl 1770688245 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4692.484340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 4692.484630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3233) [ 4694.028622] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4694.689253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4698.906032] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 20:50:40 (1770688240) [ 4699.381761] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4699.384270] LustreError: 114465:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d247dfad80 x1856696313937024/t261993005072(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:386/0 lens 504/448 e 0 to 0 dl 1770688296 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4703.663414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4704.400807] Lustre: Failing over lustre-MDT0000 [ 4704.484835] Lustre: server umount lustre-MDT0000 complete [ 4720.298667] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4720.300890] LDISKFS-fs (dm-0): recovery complete [ 4720.306121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4722.078873] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4725.749193] Lustre: 114465:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d37580d500 x1856696313937024/t261993005072(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:412/0 lens 504/2880 e 0 to 0 dl 1770688322 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4725.754547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 4725.754684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 4727.235351] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4727.937900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4731.966801] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 20:51:13 (1770688273) [ 4732.420321] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4732.422892] LustreError: 7670:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d3802ea680 x1856696313950720/t266287972368(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:419/0 lens 504/448 e 0 to 0 dl 1770688329 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4733.764683] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4737.283804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4738.312582] Lustre: Failing over lustre-MDT0000 [ 4738.439168] Lustre: server umount lustre-MDT0000 complete [ 4753.860525] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4753.862459] LDISKFS-fs (dm-0): recovery complete [ 4753.868284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.641839] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4759.539707] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d380199180 x1856696313951488/t266287972369(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:402/0 lens 392/456 e 0 to 0 dl 1770688312 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4759.545140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4759.545146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 4759.548946] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4765.048728] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 20:51:46 (1770688306) [ 4765.516877] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4765.518654] Lustre: Skipped 1 previous similar message [ 4765.520465] LustreError: 13674:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d3802e9f80 x1856696313964032/t270582939664(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:452/0 lens 504/448 e 0 to 0 dl 1770688362 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4765.529707] LustreError: 13674:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 4766.858052] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4766.860033] Lustre: Skipped 1 previous similar message [ 4770.437817] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4771.190553] Lustre: Failing over lustre-MDT0000 [ 4771.293696] Lustre: server umount lustre-MDT0000 complete [ 4786.327541] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4786.329153] LDISKFS-fs (dm-0): recovery complete [ 4786.334514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.185573] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4791.796791] Lustre: 13674:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d37da8b800 x1856696313964032/t270582939664(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:478/0 lens 504/2880 e 0 to 0 dl 1770688388 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4791.802611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4791.802827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329) [ 4796.648547] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 20:52:18 (1770688338) [ 4797.107992] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4798.447531] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4798.450693] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4798.454283] LustreError: 6486:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d343d7bc50 x1856696313976192/t274877906960(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:441/0 lens 392/456 e 0 to 0 dl 1770688351 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4802.193481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4802.922167] Lustre: Failing over lustre-MDT0000 [ 4802.986260] LustreError: 137211:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4802.988494] LustreError: 137211:0:(obd_class.h:479:obd_check_dev()) Skipped 99 previous similar messages [ 4803.010081] Lustre: server umount lustre-MDT0000 complete [ 4804.312803] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.320485] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 171 previous similar messages [ 4817.954824] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4817.956022] LDISKFS-fs (dm-0): recovery complete [ 4817.961864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.026966] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4818.031280] LustreError: Skipped 11 previous similar messages [ 4818.155118] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4818.158974] Lustre: Skipped 14 previous similar messages [ 4819.714602] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4823.547154] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d37d817100 x1856696313976192/t274877906960(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:466/0 lens 392/456 e 0 to 0 dl 1770688376 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4823.548299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 4823.548320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 4828.689094] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 20:52:50 (1770688370) [ 4829.066996] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4888.794542] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnecting [ 4888.798504] Lustre: Skipped 4 previous similar messages [ 4888.802451] Lustre: 6485:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d37580d500 x1856696313986688/t279172874255(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:575/0 lens 664/3488 e 0 to 0 dl 1770688485 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4891.498306] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 20:53:53 (1770688433) [ 4894.676546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4895.462056] Lustre: Failing over lustre-MDT0000 [ 4895.556261] Lustre: server umount lustre-MDT0000 complete [ 4900.320960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4900.324805] LustreError: Skipped 11 previous similar messages [ 4910.443445] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4910.444942] LDISKFS-fs (dm-0): recovery complete [ 4910.451282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4910.586968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.589438] Lustre: Skipped 12 previous similar messages [ 4912.118037] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4914.377179] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4914.381759] Lustre: Skipped 12 previous similar messages [ 4915.699631] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4915.703102] Lustre: Skipped 12 previous similar messages [ 4915.718486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 4915.718548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361) [ 4917.026219] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.613982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4931.455630] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 20:54:33 (1770688473) [ 4934.641775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4935.350124] Lustre: Failing over lustre-MDT0000 [ 4935.495600] Lustre: server umount lustre-MDT0000 complete [ 4950.706553] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4950.708114] LDISKFS-fs (dm-0): recovery complete [ 4950.712851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.492283] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4956.165626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3393) [ 4956.165634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 4957.636123] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4958.361388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4961.109640] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4966.966631] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 20:55:08 (1770688508) [ 4984.936629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4985.868223] Lustre: Failing over lustre-MDT0000 [ 4986.069712] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 4986.132564] Lustre: server umount lustre-MDT0000 complete [ 5001.360493] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5001.362527] LDISKFS-fs (dm-0): recovery complete [ 5001.366463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.166879] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5006.912867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5006.912882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5008.333185] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5008.983265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5026.758539] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 20:56:08 (1770688568) [ 5030.794785] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5031.594923] Lustre: Failing over lustre-MDT0000 [ 5031.798942] Lustre: server umount lustre-MDT0000 complete [ 5046.368558] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5046.370072] LDISKFS-fs (dm-0): recovery complete [ 5046.373751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.973977] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5051.877994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5051.879973] Lustre: Skipped 53 previous similar messages [ 5051.906407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 5051.906417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 5053.097283] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5053.667493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5056.752039] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5057.330798] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5059.649296] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 20:56:41 (1770688601) [ 5065.673144] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5126.373700] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5126.375636] Lustre: Skipped 1 previous similar message [ 5126.377340] LustreError: 6485:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d343d0c450 x1856696316778624/t296352743435(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:58/0 lens 66040/440 e 0 to 0 dl 1770688723 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5126.385456] LustreError: 6485:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5185.743145] Lustre: 113097:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d24afa2680 x1856696316778624/t296352743435(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:117/0 lens 66040/440 e 0 to 0 dl 1770688782 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5189.431571] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5190.062532] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 20:58:51 (1770688731) [ 5196.301726] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5197.546566] Lustre: Failing over lustre-MDT0000 [ 5197.749780] Lustre: server umount lustre-MDT0000 complete [ 5200.353213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.359145] Lustre: Skipped 55 previous similar messages [ 5213.277840] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5213.279946] LDISKFS-fs (dm-0): recovery complete [ 5213.286176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.108061] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5219.226098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5219.226485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5220.977942] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5221.716474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5226.988731] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 20:59:28 (1770688768) [ 5238.395983] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5243.195448] Lustre: Failing over lustre-OST0000 [ 5243.243956] Lustre: server umount lustre-OST0000 complete [ 5258.642718] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5258.644290] LDISKFS-fs (dm-2): recovery complete [ 5258.649681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5261.171715] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5273.150710] Lustre: Failing over lustre-OST0000 [ 5273.202501] Lustre: server umount lustre-OST0000 complete [ 5286.665459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5288.918911] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5292.425519] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5293.046069] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5328.171987] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 21:01:09 (1770688869) [ 5329.331116] Lustre: Failing over lustre-MDT0000 [ 5329.591631] Lustre: server umount lustre-MDT0000 complete [ 5330.916653] LustreError: 6467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770688873 with bad export cookie 14819532514758065645 [ 5343.350851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5345.057564] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5348.857521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5348.857555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5356.696769] Lustre: Failing over lustre-MDT0000 [ 5356.903880] Lustre: server umount lustre-MDT0000 complete [ 5370.193613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.741537] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5375.486276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5375.486340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5376.827452] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.414411] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5381.233476] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 21:02:02 (1770688922) [ 5392.769480] Lustre: Failing over lustre-OST0000 [ 5392.814202] Lustre: server umount lustre-OST0000 complete [ 5405.899098] LustreError: 8366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.908883] LustreError: 8366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 141 previous similar messages [ 5406.153550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5408.539254] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5411.943285] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5412.624413] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5416.926598] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 21:02:38 (1770688958) [ 5417.658983] Lustre: Failing over lustre-MDT0000 [ 5417.889086] LustreError: 156529:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5417.891683] LustreError: 156529:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages [ 5417.920712] Lustre: server umount lustre-MDT0000 complete [ 5421.936323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.981422] Lustre: *** cfs_fail_loc=605, val=0*** [ 5421.982961] LustreError: 157024:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc07a6c90 failed: rc = -95 [ 5421.986782] LustreError: 157024:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 5421.989101] LustreError: 157024:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 5421.991808] LustreError: 157024:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 5421.994312] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5421.996414] LustreError: 157024:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 5422.000968] Lustre: server umount lustre-MDT0000 complete [ 5422.002318] LustreError: 157024:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5425.218058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.279984] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5425.288336] LustreError: Skipped 7 previous similar messages [ 5425.427797] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5425.437102] Lustre: Skipped 10 previous similar messages [ 5427.172841] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5430.630729] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 21:02:52 (1770688972) [ 5430.790544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5430.791547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5434.152762] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5435.672443] Lustre: Failing over lustre-MDT0000 [ 5435.864426] Lustre: server umount lustre-MDT0000 complete [ 5451.382134] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5451.385445] LDISKFS-fs (dm-0): recovery complete [ 5451.389650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.981369] Lustre: *** cfs_fail_loc=707, val=0*** [ 5453.190228] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5512.415454] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5512.432026] Lustre: 159175:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5512.438854] Lustre: 159175:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 5512.588916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 5512.589282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 5514.298098] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5514.891449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5519.295882] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 21:04:20 (1770689060) [ 5543.608735] LustreError: 13674:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d36fd78700 x1856696317660160/t0(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:431/0 lens 664/0 e 0 to 0 dl 1770689096 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5543.621686] LustreError: 13674:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5554.648128] LustreError: 13674:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5554.655212] LustreError: 6486:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d37f30a680 x1856696317661824/t0(0) o35->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:442/0 lens 392/0 e 0 to 0 dl 1770689107 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5559.777902] LustreError: 42097:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d2479b9880 x1856696323574144/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:447/0 lens 544/0 e 0 to 0 dl 1770689112 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5559.784363] LustreError: 42097:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 5566.945573] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 21:05:08 (1770689108) [ 5590.699901] LustreError: 40899:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5601.728128] LustreError: 40899:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5604.878713] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 21:05:46 (1770689146) [ 5628.029269] LustreError: 6483:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d35fd6f800 x1856696317685504/t0(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:516/0 lens 576/0 e 0 to 0 dl 1770689181 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5628.039751] LustreError: 6483:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5633.040154] LustreError: 6483:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5633.047255] LustreError: 6483:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d37da29500 x1856696317686400/t0(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:521/0 lens 664/0 e 0 to 0 dl 1770689186 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5644.048120] LustreError: 6484:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5644.056317] LustreError: 6484:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d37f0ae300 x1856696317708160/t0(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:571/0 lens 664/0 e 0 to 0 dl 1770689236 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5644.065827] LustreError: 6484:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5656.498066] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 21:06:38 (1770689198) [ 5741.618277] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 21:08:03 (1770689283) [ 5764.788700] LustreError: 6485:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d37da88380 x1856696317765376/t0(0) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:652/0 lens 576/0 e 0 to 0 dl 1770689317 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5764.800353] LustreError: 6485:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 5764.804828] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5764.808016] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5765.224084] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5781.051420] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5781.054688] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5781.472118] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5781.475253] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5796.923922] LustreError: 7670:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff95d37da3c380 x1856696317784576/t0(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:685/0 lens 504/0 e 0 to 0 dl 1770689350 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5796.931463] LustreError: 7670:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 5809.106931] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 21:09:10 (1770689350) [ 5833.583714] Lustre: DEBUG MARKER: phase 2 [ 5836.535984] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 21:09:38 (1770689378) [ 5906.747857] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 21:10:48 (1770689448) [ 5907.215195] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5907.762553] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 21:10:49 (1770689449) [ 5909.509338] Lustre: DEBUG MARKER: Started rundbench load pid=133897 ... [ 5913.193478] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5914.750699] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5915.446532] Lustre: Failing over lustre-MDT0000 [ 5915.454518] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 5915.501365] LustreError: 7670:0:(ldlm_lockd.c:1452:ldlm_handle_enqueue()) ### lock on destroyed export 00000000fb13f64a ns: mdt-lustre-MDT0000_UUID lock: ffff95d252ed8c00/0xcda98e4e0f837760 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xbc1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.202.60@tcp remote: 0x7d0172d79f05243f expref: 4 pid: 7670 timeout: 0 lvb_type: 0 [ 5917.152627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5917.155754] LustreError: Skipped 9 previous similar messages [ 5917.156869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.160523] Lustre: Skipped 23 previous similar messages [ 5917.162841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5921.683718] Lustre: server umount lustre-MDT0000 complete [ 5935.703175] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5935.704423] LDISKFS-fs (dm-0): recovery complete [ 5935.707329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5935.818825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.821314] Lustre: Skipped 11 previous similar messages [ 5937.076980] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5938.377208] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5938.379866] Lustre: Skipped 11 previous similar messages [ 5941.218352] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5941.221190] Lustre: Skipped 29 previous similar messages [ 5941.357926] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5941.360507] Lustre: Skipped 11 previous similar messages [ 5941.377368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5063 to 0x2c0000401:5089) [ 5941.377402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5099 to 0x280000401:5121) [ 5942.622887] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5943.179277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5948.297095] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5949.808030] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5950.439794] Lustre: Failing over lustre-MDT0001 [ 5950.616655] Lustre: server umount lustre-MDT0001 complete [ 5964.638799] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5964.640423] LDISKFS-fs (dm-1): recovery complete [ 5964.645260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5966.025843] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5969.906653] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:449) [ 5969.906671] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:449) [ 5971.119638] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5971.712266] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5976.639337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5978.250711] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5978.938198] Lustre: Failing over lustre-MDT0000 [ 5978.943191] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 5978.945145] Lustre: Skipped 4 previous similar messages [ 5979.113979] Lustre: server umount lustre-MDT0000 complete [ 5993.140803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5993.142386] LDISKFS-fs (dm-0): recovery complete [ 5993.146441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.562250] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5998.873207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5410 to 0x280000401:5441) [ 5998.873208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5379 to 0x2c0000401:5409) [ 6000.092227] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6000.709224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6005.754706] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6007.318040] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 6008.068460] Lustre: Failing over lustre-MDT0001 [ 6008.225799] Lustre: server umount lustre-MDT0001 complete [ 6008.802091] LustreError: 114465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6008.807087] LustreError: 114465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 70 previous similar messages [ 6022.623118] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6022.625259] LDISKFS-fs (dm-1): recovery complete [ 6022.632894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6024.115516] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6028.279625] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:481) [ 6028.279639] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:481) [ 6029.526450] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6030.107674] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6055.162899] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 21:13:16 (1770689596) [ 6178.021931] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6188.778611] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6189.424402] Lustre: Failing over lustre-MDT0001 [ 6189.450037] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 6195.245679] LustreError: 173008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6195.247520] LustreError: 173008:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6195.289336] Lustre: server umount lustre-MDT0001 complete [ 6209.222863] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6209.224222] LDISKFS-fs (dm-1): recovery complete [ 6209.238150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6209.357617] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6209.360155] Lustre: Skipped 5 previous similar messages [ 6210.533846] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6217.133903] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1904 to 0x280000400:1921) [ 6217.134321] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1904 to 0x2c0000400:1921) [ 6217.187113] Lustre: 113097:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d3486f1180 x1856696352411008/t12884925333(0) o36->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:350/0 lens 488/3152 e 0 to 0 dl 1770689770 ref 1 fl Interpret:/202/0 rc 0/0 job:'tar.0' uid:0 gid:0 projid:4294967295 [ 6218.438413] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6218.984701] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6343.135887] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6353.800014] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 6354.399868] Lustre: Failing over lustre-MDT0001 [ 6354.408690] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 6354.411622] Lustre: Skipped 4 previous similar messages [ 6354.736491] Lustre: server umount lustre-MDT0001 complete [ 6368.588326] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6368.589723] LDISKFS-fs (dm-1): recovery complete [ 6368.594196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6369.898693] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6375.964270] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3372 to 0x2c0000400:3393) [ 6375.967263] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3372 to 0x280000400:3393) [ 6377.151500] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6377.716270] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6396.423569] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 21:18:58 (1770689938) [ 6519.396639] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6530.054716] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6530.643190] Lustre: Failing over lustre-MDT0000 [ 6530.649401] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 6530.887731] Lustre: server umount lustre-MDT0000 complete [ 6532.576936] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.576962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6532.580260] Lustre: Skipped 20 previous similar messages [ 6532.582194] LustreError: Skipped 4 previous similar messages [ 6544.949245] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6544.950413] LDISKFS-fs (dm-0): recovery complete [ 6544.953345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.984329] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.987384] LustreError: Skipped 3 previous similar messages [ 6545.049224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.051445] Lustre: Skipped 5 previous similar messages [ 6546.209748] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6548.680662] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6548.682724] Lustre: Skipped 5 previous similar messages [ 6550.498277] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6550.500282] Lustre: Skipped 19 previous similar messages [ 6551.520225] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6551.522734] Lustre: Skipped 5 previous similar messages [ 6551.539220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9418 to 0x280000401:9441) [ 6551.539245] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9387 to 0x2c0000401:9409) [ 6552.687787] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6553.225295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6677.080725] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6687.792979] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 6688.369596] Lustre: Failing over lustre-MDT0001 [ 6688.736805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6688.738580] Lustre: Skipped 2 previous similar messages [ 6693.970032] Lustre: server umount lustre-MDT0001 complete [ 6697.161068] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.167149] LustreError: 13674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 6707.880801] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6707.882597] LDISKFS-fs (dm-1): recovery complete [ 6707.886435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6709.140588] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6714.617557] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4430 to 0x280000400:4449) [ 6714.617597] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4430 to 0x2c0000400:4449) [ 6715.944645] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6716.527460] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6719.976132] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 21:24:21 (1770690261) [ 6843.035963] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6853.826729] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 6854.461744] Lustre: Failing over lustre-MDT0000 [ 6854.730333] LustreError: 180303:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6854.733436] LustreError: 180303:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 6854.796076] Lustre: server umount lustre-MDT0000 complete [ 6869.010466] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6869.012532] LDISKFS-fs (dm-0): recovery complete [ 6869.016272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.150648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6869.153147] Lustre: Skipped 3 previous similar messages [ 6870.534852] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6877.332439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11936 to 0x280000401:11969) [ 6877.334266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11905 to 0x2c0000401:11937) [ 6878.673217] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6879.315487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7004.159349] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7015.135068] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 7015.983458] Lustre: Failing over lustre-MDT0001 [ 7016.108136] Lustre: server umount lustre-MDT0001 complete [ 7032.130828] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7032.132835] LDISKFS-fs (dm-1): recovery complete [ 7032.144811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7034.076969] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7037.439505] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4481) [ 7037.439630] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4459 to 0x2c0000400:4481) [ 7038.764344] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7039.409547] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7043.827730] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 21:29:45 (1770690585) [ 7051.386500] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7053.156153] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7054.176087] Lustre: Failing over lustre-OST0000 [ 7054.241240] Lustre: server umount lustre-OST0000 complete [ 7069.718417] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7069.720181] LDISKFS-fs (dm-2): recovery complete [ 7069.727748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7072.301253] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7076.601367] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7077.413697] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7087.794723] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7089.571075] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7090.571253] Lustre: Failing over lustre-OST0000 [ 7090.632919] Lustre: server umount lustre-OST0000 complete [ 7107.522782] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7107.529196] LDISKFS-fs (dm-2): recovery complete [ 7107.537378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7111.072768] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7115.993885] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7116.971116] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7124.182933] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 21:31:05 (1770690665) [ 7248.536282] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7252.234575] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7263.303372] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7264.265919] Lustre: Failing over lustre-MDT0001 [ 7264.266894] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -19 [ 7264.274819] LustreError: Skipped 4 previous similar messages [ 7264.277319] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.284921] Lustre: Skipped 14 previous similar messages [ 7264.288382] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 7264.292723] Lustre: Skipped 4 previous similar messages [ 7264.435985] Lustre: server umount lustre-MDT0001 complete [ 7266.157659] Lustre: Failing over lustre-MDT0000 [ 7266.339766] Lustre: server umount lustre-MDT0000 complete [ 7282.670944] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7282.674828] LDISKFS-fs (dm-0): recovery complete [ 7282.679680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7282.741609] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7282.743210] LDISKFS-fs (dm-1): recovery complete [ 7282.750150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7282.751744] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.758717] LustreError: Skipped 1 previous similar message [ 7282.913709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7282.916168] Lustre: Skipped 5 previous similar messages [ 7283.915896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7283.922414] Lustre: Skipped 5 previous similar messages [ 7285.115272] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7285.292589] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7286.116070] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7286.118187] Lustre: Skipped 17 previous similar messages [ 7286.198239] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7286.201032] Lustre: Skipped 5 previous similar messages [ 7286.217655] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4459 to 0x2c0000400:4513) [ 7286.219468] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4513) [ 7290.760358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13243 to 0x280000401:13281) [ 7290.765791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13211 to 0x2c0000401:13249) [ 7292.677527] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7293.570895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7294.353672] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7418.927612] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7421.844952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7432.612538] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 7433.281297] Lustre: Failing over lustre-MDT0001 [ 7433.288550] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 7433.291164] Lustre: Skipped 1 previous similar message [ 7433.534859] Lustre: server umount lustre-MDT0001 complete [ 7434.720752] LustreError: 172417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.727530] LustreError: 172417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 63 previous similar messages [ 7434.832861] Lustre: Failing over lustre-MDT0000 [ 7434.991883] Lustre: server umount lustre-MDT0000 complete [ 7450.214409] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7450.217265] LDISKFS-fs (dm-0): recovery complete [ 7450.223241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.247462] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7450.252273] LDISKFS-fs (dm-1): recovery complete [ 7450.257768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7451.905526] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7452.037382] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7455.648096] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770690981/real 1770690981] req@ffff95d37ebd5f80 x1856696363980800/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770690997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7455.659722] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 7455.669254] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4459 to 0x2c0000400:4545) [ 7455.669397] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4460 to 0x280000400:4545) [ 7460.995705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13211 to 0x2c0000401:13281) [ 7460.995713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13243 to 0x280000401:13313) [ 7462.346975] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7462.964676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7463.597258] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7467.352725] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:36:49 (1770691009) [ 7470.061470] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7471.159911] Lustre: Failing over lustre-MDT0000 [ 7471.275831] LustreError: 194258:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 7471.278307] LustreError: 194258:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 7471.323948] Lustre: server umount lustre-MDT0000 complete [ 7485.977896] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7485.979672] LDISKFS-fs (dm-0): recovery complete [ 7485.983375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.124120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7486.127099] Lustre: Skipped 7 previous similar messages [ 7487.584760] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7487.696132] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7503.063936] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7503.078481] Lustre: 192592:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d24ae50700 x1856696425975552/t343597384001(343597384001) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:126/0 lens 592/3488 e 0 to 0 dl 1770691056 ref 1 fl Interpret:/606/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 7503.130234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13211 to 0x2c0000401:13313) [ 7503.130936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13315 to 0x280000401:13345) [ 7504.650246] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7505.230029] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7509.107338] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:37:30 (1770691050) [ 7512.063747] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7513.184503] Lustre: Failing over lustre-MDT0000 [ 7513.412060] Lustre: server umount lustre-MDT0000 complete [ 7528.579759] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7528.581416] LDISKFS-fs (dm-0): recovery complete [ 7528.585692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.282991] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7530.548364] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7530.550919] LustreError: 192624:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d37f3f0380 x1856696425986304/t347892350979(347892350979) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:153/0 lens 592/608 e 0 to 0 dl 1770691083 ref 1 fl Interpret:/604/0 rc 301/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 7546.083523] Lustre: lustre-MDT0000: Client 34d78110-fa36-421a-b835-6ec18dfb5b21 (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7546.090911] Lustre: 192592:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d24a10b480 x1856696425986304/t347892350979(347892350979) o101->34d78110-fa36-421a-b835-6ec18dfb5b21@192.168.202.60@tcp:169/0 lens 592/3488 e 0 to 0 dl 1770691099 ref 1 fl Interpret:/606/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 7546.145030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13315 to 0x280000401:13377) [ 7546.145096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13345) [ 7547.715210] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7548.514533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7552.672403] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:38:14 (1770691094) [ 7554.661104] Lustre: Failing over lustre-OST0000 [ 7554.715915] Lustre: server umount lustre-OST0000 complete [ 7556.318820] Lustre: Failing over lustre-MDT0000 [ 7556.402906] Lustre: server umount lustre-MDT0000 complete [ 7570.000197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.795826] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7572.838335] Lustre: lustre-MDT0000: Denying connection for new client 65b35781-f4c3-4f0e-91d7-76a74011deb7 (at 192.168.202.60@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7575.546824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13377) [ 7580.620491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7582.330430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13315 to 0x280000401:13409) [ 7582.868151] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7587.308928] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:38:49 (1770691129) [ 7587.686866] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7587.688936] LustreError: 172418:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d24dd85880 x1856696365088896/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:210/0 lens 1312/4320 e 0 to 0 dl 1770691140 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7590.363438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7591.035687] Lustre: Failing over lustre-MDT0000 [ 7591.264159] Lustre: server umount lustre-MDT0000 complete [ 7603.168148] Lustre: 192651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770691129/real 1770691129] req@ffff95d24dd84000 x1856696365088896/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1770691145 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7603.178074] Lustre: 192651:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 7605.868542] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7605.870352] LDISKFS-fs (dm-0): recovery complete [ 7605.876739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.570290] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7611.403578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13409) [ 7611.403578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13441) [ 7612.943062] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7613.538393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7617.569231] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:39:19 (1770691159) [ 7617.966187] LustreError: 8374:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d24abc8380 x1856696365118208/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:241/0 lens 2520/4320 e 0 to 0 dl 1770691171 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7621.016097] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7623.740335] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7624.420114] Lustre: Failing over lustre-MDT0001 [ 7624.423470] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 7624.425698] Lustre: Skipped 1 previous similar message [ 7629.910418] Lustre: server umount lustre-MDT0001 complete [ 7644.363911] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7644.365381] LDISKFS-fs (dm-1): recovery complete [ 7644.369688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7645.807397] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7649.775499] Lustre: 196262:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d24a3e4700 x1856696426040192/t38654707626(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:296/0 lens 560/2880 e 0 to 0 dl 1770691226 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7649.781193] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4577) [ 7649.781193] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4577) [ 7651.022424] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7651.605545] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7655.481593] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 21:39:57 (1770691197) [ 7655.892775] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7655.895261] Lustre: Skipped 1 previous similar message [ 7655.897129] LustreError: 172417:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d37ebf3800 x1856696365152128/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:278/0 lens 2520/4320 e 0 to 0 dl 1770691208 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7658.767159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7661.783567] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7662.473632] Lustre: Failing over lustre-MDT0000 [ 7662.709565] Lustre: server umount lustre-MDT0000 complete [ 7677.165331] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7677.166899] LDISKFS-fs (dm-0): recovery complete [ 7677.171888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.847687] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7682.575200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13441) [ 7682.575209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13473) [ 7683.772333] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7684.329314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7686.300376] Lustre: Failing over lustre-MDT0001 [ 7686.399068] Lustre: server umount lustre-MDT0001 complete [ 7700.625453] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7700.626602] LDISKFS-fs (dm-1): recovery complete [ 7700.631223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7702.043493] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7706.100382] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4588 to 0x2c0000400:4609) [ 7706.100383] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4588 to 0x280000400:4609) [ 7707.258386] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7707.851990] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7711.604659] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:40:53 (1770691253) [ 7717.513795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7720.183679] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7720.880941] Lustre: Failing over lustre-MDT0000 [ 7721.140121] Lustre: server umount lustre-MDT0000 complete [ 7722.413605] LustreError: 8391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691264 with bad export cookie 14819532514766983778 [ 7722.414470] Lustre: Failing over lustre-MDT0001 [ 7722.417946] LustreError: 8391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7728.097455] LustreError: 208246:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff95d24a279880 x1856696365215744/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7728.103780] LustreError: 208246:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7728.218250] Lustre: server umount lustre-MDT0001 complete [ 7743.347923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7743.350893] LDISKFS-fs (dm-0): recovery complete [ 7743.352796] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7743.355038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.355548] LDISKFS-fs (dm-1): recovery complete [ 7743.365427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7743.441863] LustreError: 209346:0:(llog.c:1643:llog_backup()) MGC192.168.202.160@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7743.444901] Lustre: 209346:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.160@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7747.040379] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d24a279880 x1856696365216896/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7748.656381] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7748.748687] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7753.272687] Lustre: 209358:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d24abcb100 x1856696426110080/t47244640328(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:401/0 lens 560/2880 e 0 to 0 dl 1770691331 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7753.285649] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4620 to 0x2c0000400:4641) [ 7753.285677] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4620 to 0x280000400:4641) [ 7753.297620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13505) [ 7753.297641] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13473) [ 7754.522488] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7755.111357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7755.609764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7759.169122] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 21:41:41 (1770691301) [ 7759.493111] LustreError: 209359:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d25073ce00 x1856696426144000/t51539607624(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:421/0 lens 560/448 e 0 to 0 dl 1770691351 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7759.502344] LustreError: 209359:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 7764.866704] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7765.494108] Lustre: Failing over lustre-MDT0000 [ 7765.724929] Lustre: server umount lustre-MDT0000 complete [ 7779.774406] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7779.776660] LDISKFS-fs (dm-0): recovery complete [ 7779.780456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7781.305464] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7784.952884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13505) [ 7784.952886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13537) [ 7786.037639] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7786.581455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7814.347507] Lustre: lustre-MDT0001: Client 65b35781-f4c3-4f0e-91d7-76a74011deb7 (at 192.168.202.60@tcp) reconnecting [ 7814.350903] Lustre: Skipped 2 previous similar messages [ 7814.354256] Lustre: 209392:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d377127480 x1856696426144000/t51539607624(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:476/0 lens 560/2880 e 0 to 0 dl 1770691406 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7816.989044] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 21:42:38 (1770691358) [ 7817.432098] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7817.433989] Lustre: Skipped 2 previous similar messages [ 7820.058369] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7820.672749] Lustre: Failing over lustre-MDT0001 [ 7820.758965] Lustre: server umount lustre-MDT0001 complete [ 7834.977611] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7834.979522] LDISKFS-fs (dm-1): recovery complete [ 7834.984707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7836.374889] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7840.241779] Lustre: 209358:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d2540e4e00 x1856696426178688/t51539607697(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:463/0 lens 560/2880 e 0 to 0 dl 1770691393 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7840.243553] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4662 to 0x280000400:4705) [ 7840.243576] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4662 to 0x2c0000400:4705) [ 7841.521489] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7842.164676] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7845.799096] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 21:43:07 (1770691387) [ 7851.668600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7854.360205] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7855.070586] Lustre: Failing over lustre-MDT0000 [ 7855.413318] Lustre: server umount lustre-MDT0000 complete [ 7862.475729] Lustre: 215275:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d375a3ca80 x1856696426209408/t55834574920(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:485/0 lens 560/2880 e 0 to 0 dl 1770691415 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7869.481662] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7869.482750] LDISKFS-fs (dm-0): recovery complete [ 7869.485181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7870.831429] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7875.075549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13537) [ 7875.075566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13569) [ 7876.189198] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7876.709828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7878.702202] Lustre: Failing over lustre-MDT0001 [ 7878.785857] Lustre: server umount lustre-MDT0001 complete [ 7880.161295] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.161310] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7880.166104] Lustre: Skipped 53 previous similar messages [ 7880.169066] LustreError: Skipped 14 previous similar messages [ 7892.564981] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7892.566255] LDISKFS-fs (dm-1): recovery complete [ 7892.570946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7892.664874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7892.667739] Lustre: Skipped 16 previous similar messages [ 7893.192849] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7893.196097] Lustre: Skipped 16 previous similar messages [ 7893.826658] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7898.082526] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7898.084387] Lustre: Skipped 55 previous similar messages [ 7898.089466] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7898.091468] Lustre: Skipped 16 previous similar messages [ 7898.105389] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4737) [ 7898.105389] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4716 to 0x2c0000400:4737) [ 7899.121963] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7899.607287] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7902.856602] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 21:44:04 (1770691444) [ 7903.171046] LustreError: 209392:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d2540e6680 x1856696426246784/t60129542216(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:526/0 lens 560/448 e 0 to 0 dl 1770691456 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7903.178270] LustreError: 209392:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 2 previous similar messages [ 7908.543829] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7911.001199] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7911.601914] Lustre: Failing over lustre-MDT0000 [ 7911.941385] Lustre: server umount lustre-MDT0000 complete [ 7913.183505] LustreError: 95802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691455 with bad export cookie 14819532514766997694 [ 7913.184948] Lustre: Failing over lustre-MDT0001 [ 7913.185132] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7913.185138] LustreError: Skipped 9 previous similar messages [ 7913.187121] LustreError: 95802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7913.285646] Lustre: server umount lustre-MDT0001 complete [ 7927.825848] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7927.827855] LDISKFS-fs (dm-0): recovery complete [ 7927.831151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.832360] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7927.836292] LDISKFS-fs (dm-1): recovery complete [ 7927.841996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7929.824185] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770691455/real 1770691455] req@ffff95d24a3e6a00 x1856696365375104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770691471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7929.833255] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 7938.016739] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d3760f5c00 x1856696365377920/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7939.650914] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7939.750316] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7940.224632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13569) [ 7940.225149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13601) [ 7944.341605] Lustre: 219949:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d3776fad80 x1856696426246784/t60129542216(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:567/0 lens 560/2880 e 0 to 0 dl 1770691497 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7944.345608] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4748 to 0x280000400:4769) [ 7944.345626] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4748 to 0x2c0000400:4769) [ 7945.458978] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7946.027246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7946.554084] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7950.097493] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:44:51 (1770691491) [ 7952.846616] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7953.439166] Lustre: Failing over lustre-MDT0001 [ 7953.448157] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 7953.449730] Lustre: Skipped 10 previous similar messages [ 7959.092508] Lustre: server umount lustre-MDT0001 complete [ 7973.109714] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7973.111301] LDISKFS-fs (dm-1): recovery complete [ 7973.117230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7974.494018] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7978.480639] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4801) [ 7978.480644] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4801) [ 7979.551064] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7980.076317] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7983.534640] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 21:45:25 (1770691525) [ 7986.329540] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7986.942313] Lustre: Failing over lustre-MDT0000 [ 7987.197519] Lustre: server umount lustre-MDT0000 complete [ 8001.210729] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8001.212446] LDISKFS-fs (dm-0): recovery complete [ 8001.216099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8002.568180] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8006.650265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13633) [ 8006.650307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13601) [ 8007.704580] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8008.198311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8011.486661] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 21:45:53 (1770691553) [ 8014.302912] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8016.896602] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8017.561469] Lustre: Failing over lustre-MDT0000 [ 8017.807753] Lustre: server umount lustre-MDT0000 complete [ 8031.723687] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8031.725009] LDISKFS-fs (dm-0): recovery complete [ 8031.727863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.019732] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8036.859043] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13633) [ 8036.859053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13665) [ 8037.952699] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8038.491898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8040.416725] Lustre: Failing over lustre-MDT0001 [ 8040.509097] Lustre: server umount lustre-MDT0001 complete [ 8041.673890] LustreError: 222929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8041.680042] LustreError: 222929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 202 previous similar messages [ 8054.424840] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8054.426502] LDISKFS-fs (dm-1): recovery complete [ 8054.430985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8055.639422] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8059.896552] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4833) [ 8059.896553] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4833) [ 8061.094212] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8061.688717] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8065.206892] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 21:46:47 (1770691607) [ 8068.179643] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8070.745236] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8071.410125] Lustre: Failing over lustre-MDT0000 [ 8071.727473] LustreError: 229198:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 8071.730568] LustreError: 229198:0:(obd_class.h:479:obd_check_dev()) Skipped 155 previous similar messages [ 8071.764178] Lustre: server umount lustre-MDT0000 complete [ 8073.072890] LustreError: 95802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691615 with bad export cookie 14819532514767008215 [ 8073.074666] Lustre: Failing over lustre-MDT0001 [ 8073.078100] LustreError: 95802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8073.080476] LustreError: 229398:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff95d347a50a00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8078.817112] LustreError: 229398:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff95d252588a80 x1856696365513600/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8078.825072] LustreError: 229398:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8078.926922] Lustre: server umount lustre-MDT0001 complete [ 8093.182272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8093.184250] LDISKFS-fs (dm-0): recovery complete [ 8093.188062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8093.192215] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8093.193689] LDISKFS-fs (dm-1): recovery complete [ 8093.197915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8098.784419] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d36fd5b480 x1856696365514368/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8098.922438] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8098.925205] Lustre: Skipped 19 previous similar messages [ 8100.428605] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8100.568568] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8103.151535] Lustre: 230512:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d247aeb100 x1856696426325632/t73014444034(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:751/0 lens 496/2888 e 0 to 0 dl 1770691681 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8103.158139] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4865) [ 8103.158399] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4865) [ 8103.163175] Lustre: 230512:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8116.452860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13697) [ 8116.452864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13665) [ 8117.711449] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8118.225580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8118.731647] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8122.028270] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 21:47:43 (1770691663) [ 8122.359859] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8122.361832] Lustre: Skipped 6 previous similar messages [ 8124.980788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8125.596202] Lustre: Failing over lustre-MDT0000 [ 8125.728029] Lustre: server umount lustre-MDT0000 complete [ 8137.928684] Lustre: lustre-MDT0001: Client 65b35781-f4c3-4f0e-91d7-76a74011deb7 (at 192.168.202.60@tcp) reconnecting [ 8137.931640] Lustre: Skipped 1 previous similar message [ 8139.429368] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8139.430523] LDISKFS-fs (dm-0): recovery complete [ 8139.432775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.687966] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8144.891699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13697) [ 8144.891708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13729) [ 8145.937092] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8146.420697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8149.729632] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 21:48:11 (1770691691) [ 8152.463145] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8153.081101] Lustre: Failing over lustre-MDT0001 [ 8153.164334] Lustre: server umount lustre-MDT0001 complete [ 8166.819755] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8166.821076] LDISKFS-fs (dm-1): recovery complete [ 8166.824454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8168.112573] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8172.016735] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4897) [ 8172.016791] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4897) [ 8173.081347] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8173.639160] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8176.732389] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 21:48:38 (1770691718) [ 8177.066988] LustreError: 230512:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff95d25258a300 x1856696426363264/t81604378626(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:45/0 lens 496/456 e 0 to 0 dl 1770691730 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8177.073844] LustreError: 230512:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 6 previous similar messages [ 8179.380839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8181.676650] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8182.226640] Lustre: Failing over lustre-MDT0000 [ 8182.367866] Lustre: server umount lustre-MDT0000 complete [ 8196.123609] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8196.125050] LDISKFS-fs (dm-0): recovery complete [ 8196.128038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.375218] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8201.727566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13729) [ 8201.727596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13761) [ 8202.768500] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8203.272261] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8205.096856] Lustre: Failing over lustre-MDT0001 [ 8205.177211] Lustre: server umount lustre-MDT0001 complete [ 8219.138693] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8219.139796] LDISKFS-fs (dm-1): recovery complete [ 8219.144341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8220.468934] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8224.758323] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4929) [ 8224.758325] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4929) [ 8225.831753] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8226.318184] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8229.596581] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 21:49:31 (1770691771) [ 8232.314526] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8234.637870] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8235.202752] Lustre: Failing over lustre-MDT0000 [ 8235.413495] Lustre: server umount lustre-MDT0000 complete [ 8236.516617] LustreError: 6467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691778 with bad export cookie 14819532514767015761 [ 8236.517751] Lustre: Failing over lustre-MDT0001 [ 8236.519993] LustreError: 6467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8236.611825] Lustre: server umount lustre-MDT0001 complete [ 8251.056972] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8251.059546] LDISKFS-fs (dm-0): recovery complete [ 8251.063288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8251.064918] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8251.067987] LDISKFS-fs (dm-1): recovery complete [ 8251.072574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8255.968142] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1770691782/real 1770691782] req@ffff95d344753b80 x1856696365646464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1770691798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8255.975169] Lustre: 3629:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 18 previous similar messages [ 8261.088461] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d37593b480 x1856696365649024/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8262.604584] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8262.693655] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8271.058754] Lustre: 241181:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95d344750000 x1856696426381312/t85899345922(0) o36->65b35781-f4c3-4f0e-91d7-76a74011deb7@192.168.202.60@tcp:139/0 lens 496/2888 e 0 to 0 dl 1770691824 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8271.065207] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4961) [ 8271.065217] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4961) [ 8271.065697] Lustre: 241181:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 3 previous similar messages [ 8276.194392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13315 to 0x2c0000401:13761) [ 8276.194396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13411 to 0x280000401:13793) [ 8277.396553] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8277.908716] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8278.395510] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8281.557537] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 21:50:23 (1770691823) [ 8282.117510] Lustre: 242385:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 65b35781-f4c3-4f0e-91d7-76a74011deb7 at adminstrative request [ 8286.549923] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 21:50:28 (1770691828) [ 8288.335066] Lustre: Failing over lustre-MDT0000 [ 8288.568874] Lustre: server umount lustre-MDT0000 complete [ 8301.102237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.355981] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8306.703536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13845 to 0x280000401:13889) [ 8306.703588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13812 to 0x2c0000401:13857) [ 8307.859427] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8308.361947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8311.482715] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 21:50:53 (1770691853) [ 8314.905970] Lustre: Failing over lustre-OST0000 [ 8314.966322] Lustre: server umount lustre-OST0000 complete [ 8327.725728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8329.816757] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8333.005680] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8333.614327] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8337.139547] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 21:51:18 (1770691878) [ 8338.501417] Lustre: Failing over lustre-MDT0000 [ 8338.749601] Lustre: server umount lustre-MDT0000 complete [ 8340.963529] LustreError: 8391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691883 with bad export cookie 14819532514767028340 [ 8341.235572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.656529] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8343.376234] Lustre: lustre-MDT0000: Denying connection for new client ddf8120d-053e-4baa-8dbe-136d25f3a672 (at 192.168.202.60@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8343.381401] Lustre: Skipped 1 previous similar message [ 8346.618525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13845 to 0x280000401:13921) [ 8346.618531] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13812 to 0x2c0000401:13889) [ 8351.051200] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 21:51:32 (1770691892) [ 8353.844658] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8354.740033] Lustre: Failing over lustre-OST0000 [ 8354.771243] Lustre: server umount lustre-OST0000 complete [ 8368.872261] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8368.873979] LDISKFS-fs (dm-2): recovery complete [ 8368.877799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8370.851433] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8374.058245] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8374.621952] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8378.291196] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 21:52:00 (1770691920) [ 8381.009276] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8382.810190] Lustre: Failing over lustre-OST0000 [ 8382.840420] Lustre: server umount lustre-OST0000 complete [ 8396.591520] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8396.592603] LDISKFS-fs (dm-2): recovery complete [ 8396.595368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8397.827639] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.60@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:13923 extent [0-4194303]: client csum 2fd8b1e4, server csum 8235c18e [ 8398.428232] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8401.392948] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8401.946022] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8405.151131] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 21:52:27 (1770691947) [ 8407.446723] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8409.753180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8411.523901] Lustre: Failing over lustre-MDT0000 [ 8411.735775] Lustre: server umount lustre-MDT0000 complete [ 8412.886499] LustreError: 95802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770691954 with bad export cookie 14819532514767042578 [ 8412.888146] Lustre: Failing over lustre-OST0000 [ 8412.889872] LustreError: 95802:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8412.933322] Lustre: server umount lustre-OST0000 complete [ 8427.116076] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8427.117289] LDISKFS-fs (dm-0): recovery complete [ 8427.119943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.728407] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95d37f3f3b80 x1856696365791744/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8439.205149] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8442.949315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13812 to 0x2c0000401:13921) [ 8453.478393] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8453.480023] LDISKFS-fs (dm-2): recovery complete [ 8453.483213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8455.399324] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8461.119465] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 21:53:22 (1770692002) [ 8465.878857] Lustre: Failing over lustre-OST0000 [ 8465.933057] Lustre: server umount lustre-OST0000 complete [ 8467.224693] Lustre: Failing over lustre-MDT0000 [ 8467.573055] Lustre: server umount lustre-MDT0000 complete [ 8480.236684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.528784] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8485.366615] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13812 to 0x2c0000401:13953) [ 8491.241888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8493.216404] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8493.914433] Lustre: lustre-OST0000: Denying connection for new client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 8493.920748] Lustre: Skipped 1 previous similar message [ 8499.401054] Lustre: lustre-OST0000: Denying connection for new client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 8509.641071] Lustre: lustre-OST0000: Denying connection for new client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 8509.646250] Lustre: Skipped 1 previous similar message [ 8530.120780] Lustre: lustre-OST0000: Denying connection for new client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 8530.125901] Lustre: Skipped 3 previous similar messages [ 8562.500182] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8562.502290] Lustre: 255409:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client ddf8120d-053e-4baa-8dbe-136d25f3a672@ [ 8562.506204] Lustre: 255409:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8562.508762] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8562.512985] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8562.513134] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 8562.514380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13972 to 0x280000401:13993) [ 8562.516036] Lustre: Skipped 70 previous similar messages [ 8562.519355] Lustre: Skipped 22 previous similar messages [ 8567.318693] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 8571.885028] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 8574.109532] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 21:55:15 (1770692115) [ 8575.533284] Lustre: Failing over lustre-OST0001 [ 8575.584495] Lustre: server umount lustre-OST0001 complete [ 8577.504655] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8577.506748] LustreError: Skipped 16 previous similar messages [ 8577.508106] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8577.511952] Lustre: Skipped 64 previous similar messages [ 8588.761747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8588.817730] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8588.820180] Lustre: Skipped 23 previous similar messages [ 8590.126467] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8590.129555] Lustre: Skipped 23 previous similar messages [ 8590.819990] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8594.900193] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 21:55:36 (1770692136) [ 8596.087521] Lustre: Failing over lustre-OST0000 [ 8596.130573] Lustre: server umount lustre-OST0000 complete [ 8608.815575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8610.737631] LustreError: 258379:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8610.740575] LustreError: 258379:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 76 previous similar messages [ 8610.808875] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8616.416144] Lustre: *** cfs_fail_loc=715, val=40*** [ 8626.144601] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:25 [ 8626.148655] Lustre: Skipped 1 previous similar message [ 8627.417467] Lustre: lustre-OST0000: Client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 8632.288215] Lustre: *** cfs_fail_loc=715, val=40*** [ 8632.289673] Lustre: Skipped 2 previous similar messages [ 8633.312144] Lustre: *** cfs_fail_loc=715, val=40*** [ 8633.313768] Lustre: Skipped 1 previous similar message [ 8642.528638] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 8642.532354] Lustre: Skipped 1 previous similar message [ 8648.672175] Lustre: *** cfs_fail_loc=715, val=40*** [ 8650.784121] LustreError: 258379:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8650.786628] LustreError: 258379:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 74 previous similar messages [ 8652.115785] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8652.658581] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8656.221290] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 21:56:38 (1770692198) [ 8657.602132] Lustre: Failing over lustre-MDT0000 [ 8657.893451] Lustre: server umount lustre-MDT0000 complete [ 8658.121610] LustreError: 241150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.127445] LustreError: 241150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 196 previous similar messages [ 8670.658276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.690639] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.693648] LustreError: Skipped 10 previous similar messages [ 8672.043251] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8675.814259] LustreError: 259820:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8679.904195] Lustre: *** cfs_fail_loc=715, val=80*** [ 8679.905453] Lustre: Skipped 2 previous similar messages [ 8688.840889] Lustre: lustre-MDT0000: Client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 8688.844032] Lustre: Skipped 1 previous similar message [ 8691.168557] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8695.264124] Lustre: *** cfs_fail_loc=715, val=80*** [ 8695.265419] Lustre: Skipped 1 previous similar message [ 8705.224611] Lustre: lustre-MDT0000: Client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 8707.552502] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8711.648150] Lustre: *** cfs_fail_loc=715, val=80*** [ 8711.650093] Lustre: Skipped 1 previous similar message [ 8721.608956] Lustre: lustre-MDT0000: Client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8722.912871] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8739.296471] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8745.440169] Lustre: *** cfs_fail_loc=715, val=80*** [ 8745.441425] Lustre: Skipped 4 previous similar messages [ 8753.352787] Lustre: lustre-MDT0000: Client 559c2680-00e4-42f5-be1c-4cfcd22a2aaf (at 192.168.202.60@tcp) reconnected, waiting for 2 clients in recovery for 0:23 [ 8753.357301] Lustre: Skipped 1 previous similar message [ 8755.680543] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8755.896098] LustreError: 259820:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8755.901307] Lustre: 259820:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 8755.905407] LustreError: dumping log to /tmp/lustre-log.1770692298.259820 [ 8755.979844] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13966 to 0x2c0000401:13985) [ 8755.979920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14007 to 0x280000401:14025) [ 8757.275854] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8757.805610] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8761.326724] Lustre: DEBUG MARKER: == replay-single test complete, duration 8483 sec ======== 21:58:23 (1770692303) [ 8761.837309] Lustre: DEBUG MARKER: === replay-single: start cleanup 21:58:23 (1770692303) === [ 8764.165909] Lustre: DEBUG MARKER: === replay-single: finish cleanup 21:58:25 (1770692305) === [ 8764.822050] Lustre: Failing over lustre-MDT0000 [ 8764.896832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8764.898650] Lustre: Skipped 17 previous similar messages [ 8764.962301] LustreError: 260775:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 8764.965583] LustreError: 260775:0:(obd_class.h:479:obd_check_dev()) Skipped 131 previous similar messages [ 8764.993922] Lustre: server umount lustre-MDT0000 complete [ 8780.304314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.498852] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8780.501104] Lustre: Skipped 19 previous similar messages [ 8781.909952] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8785.914711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14007 to 0x280000401:14057) [ 8785.914846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13966 to 0x2c0000401:14017) [ 8787.072782] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8787.645876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8800.351985] Lustre: server umount lustre-MDT0000 complete [ 8803.145813] LustreError: 6467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1770692345 with bad export cookie 14819532514767075548 [ 8803.151539] LustreError: 6467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8803.271716] Lustre: server umount lustre-MDT0001 complete [ 8816.640767] Lustre: server umount lustre-OST0000 complete [ 8829.658283] Lustre: server umount lustre-OST0001 complete [ 8835.118802] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 8836.223637] Key type lgssc unregistered [ 8836.359259] LNet: 264395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8836.361804] LNetError: 264395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8836.371342] LNet: Removed LNI 192.168.202.160@tcp [ 8836.725117] Key type .llcrypt unregistered [ 8836.726310] Key type ._llcrypt unregistered