[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487944335 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007377] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012106] LSM: Security Framework initializing [ 0.013056] Yama: becoming mindful. [ 0.014046] SELinux: Initializing. [ 0.015093] *** VALIDATE selinux *** [ 0.023823] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027821] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] *** VALIDATE tmpfs *** [ 0.032223] *** VALIDATE proc *** [ 0.034010] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037297] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040034] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.043757] debug: unmapping init [mem 0xffffffff8f659000-0xffffffff8f660fff] [ 0.045867] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046733] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049018] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054336] rcu: Hierarchical SRCU implementation. [ 0.056736] smp: Bringing up secondary CPUs ... [ 0.057787] x86: Booting SMP configuration: [ 0.058035] .... node #0, CPUs: #1 #2 #3 [ 0.062094] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.188024] node 0 deferred pages initialised in 121ms [ 0.191125] devtmpfs: initialized [ 0.192322] x86/mm: Memory block size: 128MB [ 0.195950] gcov: version magic: 0x41383552 [ 0.199314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.204120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.206503] pinctrl core: initialized pinctrl subsystem [ 0.208316] [ 0.208766] ************************************************************* [ 0.211014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214017] ** ** [ 0.217015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.220019] ** ** [ 0.222015] ** This means that this kernel is built to expose internal ** [ 0.225015] ** IOMMU data structures, which may compromise security on ** [ 0.227013] ** your system. ** [ 0.229010] ** ** [ 0.232014] ** If you see this message and you are not debugging the ** [ 0.234015] ** kernel, report this immediately to your vendor! ** [ 0.237016] ** ** [ 0.239012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241017] ************************************************************* [ 0.243578] NET: Registered protocol family 16 [ 0.245355] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.247041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.249059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.251180] cpuidle: using governor menu [ 0.252596] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.255073] PCI: Using configuration type 1 for base access [ 0.258146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.268133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.270025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.273177] cryptd: max_cpu_qlen set to 1000 [ 0.275777] ACPI: Added _OSI(Module Device) [ 0.276058] ACPI: Added _OSI(Processor Device) [ 0.277019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.279017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.283744] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.291023] ACPI: Interpreter enabled [ 0.293068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.294018] ACPI: Using IOAPIC for interrupt routing [ 0.295189] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.299430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.312038] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.314095] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317042] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.322104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.327259] acpiphp: Slot [2] registered [ 0.329121] acpiphp: Slot [5] registered [ 0.330105] acpiphp: Slot [6] registered [ 0.331085] acpiphp: Slot [7] registered [ 0.332000] acpiphp: Slot [8] registered [ 0.332265] acpiphp: Slot [9] registered [ 0.334233] acpiphp: Slot [10] registered [ 0.336270] acpiphp: Slot [3] registered [ 0.338164] acpiphp: Slot [4] registered [ 0.340173] acpiphp: Slot [11] registered [ 0.341176] acpiphp: Slot [12] registered [ 0.343197] acpiphp: Slot [13] registered [ 0.345163] acpiphp: Slot [14] registered [ 0.346201] acpiphp: Slot [15] registered [ 0.348180] acpiphp: Slot [16] registered [ 0.350162] acpiphp: Slot [17] registered [ 0.352163] acpiphp: Slot [18] registered [ 0.354223] acpiphp: Slot [19] registered [ 0.356182] acpiphp: Slot [20] registered [ 0.357128] acpiphp: Slot [21] registered [ 0.359117] acpiphp: Slot [22] registered [ 0.360151] acpiphp: Slot [23] registered [ 0.362130] acpiphp: Slot [24] registered [ 0.365150] acpiphp: Slot [25] registered [ 0.367115] acpiphp: Slot [26] registered [ 0.369129] acpiphp: Slot [27] registered [ 0.370139] acpiphp: Slot [28] registered [ 0.372137] acpiphp: Slot [29] registered [ 0.373136] acpiphp: Slot [30] registered [ 0.375103] acpiphp: Slot [31] registered [ 0.376067] PCI host bridge to bus 0000:00 [ 0.378026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.380028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.382029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.384028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.387038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.390074] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.393274] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.397327] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.400510] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.412027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.417047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.420047] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.423033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.427034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.432826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.435051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.437861] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.458021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.464014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.472998] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.481018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.487020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.506018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.519358] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.528017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.537016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563722] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.574030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.582026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.605026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.618163] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.628015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.634016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.654016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.663976] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.672026] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.706159] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.719886] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.733015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.746019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.760881] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.763498] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.766499] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.768476] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.771295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.778208] iommu: Default domain type: Passthrough [ 0.779623] SCSI subsystem initialized [ 0.780223] ACPI: bus type USB registered [ 0.781164] usbcore: registered new interface driver usbfs [ 0.784198] usbcore: registered new interface driver hub [ 0.787139] usbcore: registered new device driver usb [ 0.790196] pps_core: LinuxPPS API ver. 1 registered [ 0.793019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.798080] PTP clock support registered [ 0.800111] EDAC MC: Ver: 3.0.0 [ 0.803198] PCI: Using ACPI for IRQ routing [ 0.804496] NetLabel: Initializing [ 0.805012] NetLabel: domain hash size = 128 [ 0.807008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.809101] NetLabel: unlabeled traffic allowed by default [ 0.812085] vgaarb: loaded [ 0.814286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.816016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.822354] clocksource: Switched to clocksource kvm-clock [ 0.933259] VFS: Disk quotas dquot_6.6.0 [ 0.935262] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.938563] *** VALIDATE ramfs *** [ 0.939930] *** VALIDATE hugetlbfs *** [ 0.941976] pnp: PnP ACPI init [ 0.944669] pnp: PnP ACPI: found 6 devices [ 0.965535] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.968993] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.971516] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.974153] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.976632] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.979729] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.983053] NET: Registered protocol family 2 [ 0.985944] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.993343] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.997977] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005183] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.009059] TCP: Hash tables configured (established 65536 bind 65536) [ 1.012712] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.016576] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.019856] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.023446] NET: Registered protocol family 1 [ 1.027707] RPC: Registered named UNIX socket transport module. [ 1.030246] RPC: Registered udp transport module. [ 1.032344] RPC: Registered tcp transport module. [ 1.034558] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.037426] NET: Registered protocol family 44 [ 1.039667] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042125] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.044755] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.047590] PCI: CLS 0 bytes, default 64 [ 1.050166] Unpacking initramfs... [ 2.474254] debug: unmapping init [mem 0xffff96893cc54000-0xffff96893ffbffff] [ 2.478575] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.480901] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.483949] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.981806] Initialise system trusted keyrings [ 2.984236] Key type blacklist registered [ 2.986488] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.996859] zbud: loaded [ 3.000225] *** VALIDATE nfs *** [ 3.001262] *** VALIDATE nfs4 *** [ 3.002958] pstore: using deflate compression [ 3.014565] Platform Keyring initialized [ 3.113298] NET: Registered protocol family 38 [ 3.115110] Key type asymmetric registered [ 3.116430] Asymmetric key parser 'x509' registered [ 3.118021] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.120801] io scheduler mq-deadline registered [ 3.122318] io scheduler kyber registered [ 3.123818] io scheduler bfq registered [ 3.125585] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.128392] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.131101] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.133981] ACPI: Power Button [PWRF] [ 3.139646] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.146311] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.159863] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.168667] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.203810] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233433] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.263252] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.268259] Non-volatile memory driver v1.3 [ 3.269699] Linux agpgart interface v0.103 [ 3.299101] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.302488] vda: detected capacity change from 0 to 34816000 [ 3.317804] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.320317] vdb: detected capacity change from 0 to 1073741824 [ 3.335605] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.338355] vdc: detected capacity change from 0 to 2621440000 [ 3.352123] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.355511] vdd: detected capacity change from 0 to 2621440000 [ 3.371046] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.373820] vde: detected capacity change from 0 to 4294967296 [ 3.391881] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.397174] vdf: detected capacity change from 0 to 4294967296 [ 3.407892] libphy: Fixed MDIO Bus: probed [ 3.416781] usbcore: registered new interface driver usbserial_generic [ 3.419461] usbserial: USB Serial support registered for generic [ 3.421162] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.425114] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.426832] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.428831] mousedev: PS/2 mouse device common for all mice [ 3.432519] rtc_cmos 00:05: RTC can wake from S4 [ 3.434404] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.438724] rtc_cmos 00:05: registered as rtc0 [ 3.440535] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.441146] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.446370] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.446529] intel_pstate: CPU model not supported [ 3.451309] hid: raw HID events driver (C) Jiri Kosina [ 3.452926] usbcore: registered new interface driver usbhid [ 3.455068] usbhid: USB HID core driver [ 3.456820] drop_monitor: Initializing network drop monitor service [ 3.459424] Initializing XFRM netlink socket [ 3.461401] NET: Registered protocol family 10 [ 3.464819] Segment Routing with IPv6 [ 3.466243] NET: Registered protocol family 17 [ 3.468481] mpls_gso: MPLS GSO support [ 3.476611] RAS: Correctable Errors collector initialized. [ 3.479736] AVX version of gcm_enc/dec engaged. [ 3.482449] AES CTR mode by8 optimization enabled [ 3.557618] sched_clock: Marking stable (3557587965, 0)->(4507079901, -949491936) [ 3.561749] registered taskstats version 1 [ 3.564170] Loading compiled-in X.509 certificates [ 3.566711] zswap: loaded using pool lzo/zbud [ 3.592832] Key type big_key registered [ 3.605361] Key type encrypted registered [ 3.607142] ima: No TPM chip found, activating TPM-bypass! [ 3.609665] ima: Allocated hash algorithm: sha1 [ 3.612066] ima: No architecture policies found [ 3.614352] evm: Initialising EVM extended attributes: [ 3.617329] evm: security.selinux [ 3.619029] evm: security.ima [ 3.620587] evm: security.capability [ 3.622322] evm: HMAC attrs: 0x1 [ 3.625878] rtc_cmos 00:05: setting system clock to 2026-04-14 21:01:07 UTC (1776200467) [ 3.633558] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 3.637910] debug: unmapping init [mem 0xffffffff8f382000-0xffffffff8f658fff] [ 3.647089] Write protecting the kernel read-only data: 28672k [ 3.651423] debug: unmapping init [mem 0xffffffff8da03000-0xffffffff8dbfffff] [ 3.655356] debug: unmapping init [mem 0xffffffff8e314000-0xffffffff8e3fffff] [ 3.688841] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.701975] systemd[1]: Detected virtualization kvm. [ 3.704604] systemd[1]: Detected architecture x86-64. [ 3.707069] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.758103] systemd[1]: No hostname configured. [ 3.759644] systemd[1]: Set hostname to . [ 3.761418] random: systemd: uninitialized urandom read (16 bytes read) [ 3.763629] systemd[1]: Initializing machine ID from random generator. [ 3.819559] random: ln: uninitialized urandom read (6 bytes read) [ 3.909450] random: systemd: uninitialized urandom read (16 bytes read) [ 3.911427] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.916975] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.928185] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.550661] device-mapper: uevent: version 1.0.3 [ 4.552663] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 5.130151] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.361389] virtio_net virtio0 ens2: renamed from eth0 [ 5.421235] scsi host0: ata_piix [ 5.427880] scsi host1: ata_piix [ 5.429386] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.432359] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.172454] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.128063] random: crng init done [ 10.129598] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.565381] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.685730] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.971262] SELinux: Disabled at runtime. [ 12.033772] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.044446] systemd[1]: Detected virtualization kvm. [ 12.046877] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.574776] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.578536] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.583802] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.589392] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.593964] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.602416] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.610315] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ 12.653458] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.077857] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.391070] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.418985] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.546379] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.561708] EDAC sbridge: Ver: 1.1.2 [ 15.131195] Key type dns_resolver registered [ 15.480182] NFS: Registering the id_resolver key type [ 15.483387] Key type id_resolver registered [ 15.485262] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg448-server login: [ 40.010689] libcfs: loading out-of-tree module taints kernel. [ 40.032455] alg: No test for adler32 (adler32-zlib) [ 40.784368] Key type ._llcrypt registered [ 40.785789] Key type .llcrypt registered [ 40.840779] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 49.135918] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 49.737651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.373175] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 51.266088] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 51.270666] LNet: Accept secure, port 988 [ 53.191242] Key type lgssc registered [ 54.813901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.503954] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 76.723980] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.202466] hrtimer: interrupt took 5046793 ns [ 88.141752] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.135744] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.192489] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.971837] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 138.674638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.779503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.116423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 140.150693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 140.264214] Lustre: lustre-MDT0000: new disk, initializing [ 140.430163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.455567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.860342] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 159.033071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 159.173629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.316386] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 159.354206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 159.362664] Lustre: Skipped 1 previous similar message [ 159.495621] Lustre: lustre-MDT0001: new disk, initializing [ 159.649378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.710527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.729483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.571579] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 179.301585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.419079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.431613] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 179.828898] Lustre: lustre-OST0000: new disk, initializing [ 179.838670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 179.951889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.019551] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 188.479486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.517269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.833250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.916356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.937361] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 199.049974] Lustre: lustre-OST0001: new disk, initializing [ 199.065340] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 199.164970] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.245903] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 207.443994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.465299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.347088] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.898393] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 236.572709] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.913583] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing yml_node [ 246.702114] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 249.423357] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 252.079481] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 253.970397] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 14 17:05:15 EDT 2026 [ 262.329079] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 263.809185] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 282.592223] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 282.600710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 282.612704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.641067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.647238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.807786] Lustre: server umount lustre-MDT0000 complete [ 289.760233] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.787940] LustreError: Skipped 3 previous similar messages [ 290.094432] LustreError: 6220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200753 with bad export cookie 17535090620046334759 [ 290.097425] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 290.107959] LustreError: 6220:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 294.880724] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.883619] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.914358] LustreError: Skipped 1 previous similar message [ 294.935180] Lustre: Skipped 2 previous similar messages [ 294.938881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.938890] Lustre: Skipped 3 previous similar messages [ 295.908698] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.933047] LustreError: Skipped 1 previous similar message [ 296.574546] Lustre: server umount lustre-MDT0001 complete [ 301.857809] Lustre: server umount lustre-OST0000 complete [ 307.108548] Lustre: server umount lustre-OST0001 complete [ 317.966971] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 330.813505] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 343.440893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 352.934971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 360.284751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 367.527029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.262980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.328193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.664234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 375.704800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 375.790381] Lustre: lustre-MDT0000: new disk, initializing [ 375.930573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.978752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.880746] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 393.566142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 393.631808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.731933] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 393.797242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 393.801063] Lustre: Skipped 1 previous similar message [ 393.891256] Lustre: lustre-MDT0001: new disk, initializing [ 394.018405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.053473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.072270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 399.635392] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 409.940691] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 416.982377] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 424.082604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 424.167932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.175614] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 424.391937] Lustre: lustre-OST0000: new disk, initializing [ 424.400610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 424.513719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.376270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.391399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.172776] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 440.315978] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 445.467602] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 445.734330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 450.764807] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 451.099316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.186524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.188563] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.190603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.190612] Lustre: Skipped 3 previous similar messages [ 457.218684] Lustre: Skipped 2 previous similar messages [ 459.736366] Lustre: server umount lustre-OST0000 complete [ 465.891609] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 465.898409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.919787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.927094] Lustre: Skipped 1 previous similar message [ 470.191539] Lustre: server umount lustre-MDT0000 complete [ 472.545435] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.564499] LustreError: Skipped 1 previous similar message [ 474.531789] LustreError: 14317:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776200938 with bad export cookie 17535090620046336215 [ 474.536446] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.554642] LustreError: 14317:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 481.182430] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 17:09:02 (1776200942) [ 488.807745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.421296] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.435669] LustreError: Skipped 2 previous similar messages [ 489.517486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.716507] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 499.688303] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.704939] LustreError: Skipped 1 previous similar message [ 503.683189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.885746] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 517.453944] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.257810] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 530.720794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.152771] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 544.296376] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 548.179341] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 555.489196] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 555.494431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.514280] Lustre: Skipped 1 previous similar message [ 555.526854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 555.529209] Lustre: Skipped 2 previous similar messages [ 561.162317] Lustre: server umount lustre-MDT0000 complete [ 561.167543] Lustre: Skipped 1 previous similar message [ 564.708053] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.731253] LustreError: Skipped 2 previous similar messages [ 565.287666] LustreError: 18711:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201029 with bad export cookie 17535090620046337062 [ 565.290418] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 565.298971] LustreError: 18711:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 576.480201] Lustre: 21839:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201034/real 1776201034] req@00000000c78d05c6 x1862481231302592/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201040 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 598.488499] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 607.363764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.738553] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.791145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.797682] Lustre: Skipped 2 previous similar messages [ 612.805840] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 620.638548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.078769] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 633.214717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 637.898742] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 646.669364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 646.797300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 646.806361] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 646.971684] Lustre: lustre-OST0001: new disk, initializing [ 646.978056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 648.931978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 648.949360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 652.222967] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 661.840969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 666.816407] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 685.027824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.031248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.054669] Lustre: Skipped 4 previous similar messages [ 685.066849] Lustre: Skipped 6 previous similar messages [ 690.073262] Lustre: server umount lustre-MDT0000 complete [ 690.075041] Lustre: Skipped 2 previous similar messages [ 690.154439] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.175598] LustreError: Skipped 5 previous similar messages [ 694.319056] LustreError: 23206:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201158 with bad export cookie 17535090620046337916 [ 694.321359] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.336828] LustreError: 23206:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.367162] Lustre: 3333:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201159/real 1776201159] req@00000000594e16d4 x1862481231334080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201166 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 704.991209] Lustre: 27041:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201162/real 1776201162] req@000000001815cbe3 x1862481231334208/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201168 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 707.423220] Lustre: 3331:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201164/real 1776201164] req@0000000099c4a9f3 x1862481231334336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201171 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 707.438842] Lustre: 3331:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 726.234306] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 735.924845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.448559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.455648] Lustre: Skipped 3 previous similar messages [ 740.638819] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 748.910522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.396177] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 763.535256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 769.020936] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 778.527921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 784.790682] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 793.608747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 797.629475] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 797.633882] Lustre: Skipped 1 previous similar message [ 809.958865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.977080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 809.994378] Lustre: Skipped 5 previous similar messages [ 810.008058] Lustre: Skipped 3 previous similar messages [ 815.708842] Lustre: server umount lustre-MDT0000 complete [ 815.715323] Lustre: Skipped 3 previous similar messages [ 819.417523] LustreError: 28455:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201283 with bad export cookie 17535090620046339162 [ 819.433121] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.435395] LustreError: 28455:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 827.295195] Lustre: 3331:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201284/real 1776201284] req@00000000315c1835 x1862481231361216/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201291 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 843.151698] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 17:15:05 (1776201305) [ 869.584732] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 880.229274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.642903] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.657225] LustreError: Skipped 10 previous similar messages [ 885.501573] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 894.208360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.260890] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 907.890865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 912.911945] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 922.162161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 927.046276] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 935.147963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 939.311551] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 950.757619] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 950.769906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.789762] Lustre: Skipped 1 previous similar message [ 950.793692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.800612] Lustre: Skipped 4 previous similar messages [ 957.307528] Lustre: server umount lustre-MDT0000 complete [ 957.311517] Lustre: Skipped 3 previous similar messages [ 961.406118] LustreError: 34296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201425 with bad export cookie 17535090620046340233 [ 961.408345] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.419748] LustreError: 34296:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 970.655298] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201427/real 1776201427] req@00000000d9c0324d x1862481231388352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201434 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 970.683831] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 984.179517] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 17:17:25 (1776201445) [ 1000.818108] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1011.914653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.292490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.304212] Lustre: Skipped 7 previous similar messages [ 1017.200876] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1026.961601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.057920] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1039.799503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.942376] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1053.266948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.308631] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1066.382271] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1069.820732] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1078.757779] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1078.773632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.786295] Lustre: Skipped 4 previous similar messages [ 1079.267608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.280400] Lustre: Skipped 8 previous similar messages [ 1089.153517] LustreError: 40309:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201553 with bad export cookie 17535090620046341304 [ 1089.154376] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.162400] LustreError: 40309:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1103.949967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.913973] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0xf3592803982574b8 to 0xf359280398257918 [ 1106.932539] Lustre: MGC192.168.204.148@tcp: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 1107.334835] BUG: spinlock bad magic on CPU#3, lctl/43425 [ 1107.339481] lock: 0xffff9689bf6ac500, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1107.342970] CPU: 3 PID: 43425 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1107.346828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 1107.352875] Call Trace: [ 1107.354232] ? dump_stack+0xbb/0x10e [ 1107.358602] ? spin_dump.cold.3+0x32/0x65 [ 1107.359930] ? do_raw_spin_lock+0xce/0x190 [ 1107.361745] ? _raw_spin_lock+0x21/0x30 [ 1107.366294] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 1107.370871] ? __seccomp_filter+0x4b/0x680 [ 1107.374460] ? full_proxy_write+0x5e/0xa0 [ 1107.377803] ? __vfs_write+0x1c/0x60 [ 1107.380375] ? vfs_write+0xd8/0x2b0 [ 1107.381702] ? ksys_write+0x66/0x120 [ 1107.383158] ? __x64_sys_write+0x1e/0x30 [ 1107.393219] ? do_syscall_64+0xc1/0x440 [ 1107.395711] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1112.518294] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1120.748330] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1121.614805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.358486] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1127.406192] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1127.412474] Lustre: Skipped 1 previous similar message [ 1134.039704] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1135.709211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1142.185557] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1144.319223] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1152.993117] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1158.124576] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.148460] LustreError: Skipped 43 previous similar messages [ 1161.552595] LustreError: 39485:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201625 with bad export cookie 17535090620046342424 [ 1161.570041] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1161.576143] LustreError: 39485:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1170.271201] Lustre: 3331:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201627/real 1776201627] req@0000000012b21d4f x1862481231434368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776201634 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1170.313178] Lustre: 3331:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1186.169248] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 17:20:47 (1776201647) [ 1205.276341] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1216.509754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.690864] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1231.041401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.098329] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1246.403184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.349864] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1261.650366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.845674] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1277.542504] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1303.016461] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1308.864796] Lustre: server umount lustre-MDT0000 complete [ 1308.871264] Lustre: Skipped 9 previous similar messages [ 1313.243772] LustreError: 47255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776201777 with bad export cookie 17535090620046343236 [ 1313.249636] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.291388] LustreError: 47255:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1336.658229] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 17:23:17 (1776201797) [ 1339.980863] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1342.126699] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 17:23:23 (1776201803) [ 1358.017595] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 1367.708640] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1378.303494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1385.364267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1392.460246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1392.525723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.742859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1392.751274] Lustre: Skipped 3 previous similar messages [ 1392.821515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1392.905913] Lustre: lustre-MDT0000: new disk, initializing [ 1392.986846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1398.252109] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1409.223703] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1416.624858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1416.675650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1416.688463] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1416.980976] Lustre: lustre-OST0000: new disk, initializing [ 1416.993996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1417.016453] Lustre: Skipped 1 previous similar message [ 1422.263982] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1432.383381] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1434.836239] Lustre: DEBUG MARKER: create test files [ 1464.801410] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1464.814401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.832036] Lustre: Skipped 16 previous similar messages [ 1464.839380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1464.845452] Lustre: Skipped 25 previous similar messages [ 1480.671205] Lustre: 55596:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776201938/real 1776201938] req@0000000080bd1219 x1862481231475008/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776201944 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1480.707492] Lustre: 55596:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1494.754757] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1498.923343] Key type lgssc unregistered [ 1499.302794] LNet: 56200:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.318932] LNet: Removed LNI 192.168.204.148@tcp [ 1500.376876] Key type .llcrypt unregistered [ 1500.380349] Key type ._llcrypt unregistered [ 1528.412498] alg: No test for adler32 (adler32-zlib) [ 1529.165520] Key type ._llcrypt registered [ 1529.169557] Key type .llcrypt registered [ 1529.299662] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1530.492568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.815250] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 1531.070132] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1531.080475] LNet: Accept secure, port 988 [ 1532.743129] Key type lgssc registered [ 1533.949656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.521161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.025768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.826233] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1555.524378] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1563.114278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1563.556031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.708722] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1573.811844] Lustre: lustre-OST0000: deleting orphan objects from 0x0:102 to 0x0:129 [ 1573.920288] Lustre: 59045:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1573.935355] Lustre: 59045:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1573.946205] Lustre: 59045:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1573.950879] Lustre: 59045:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1573.958723] Lustre: 59045:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1573.969873] Lustre: 59045:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1575.324707] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1580.479928] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1586.187091] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1594.339577] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1594.378197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.393534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1599.460074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1599.584327] Lustre: server umount lustre-OST0000 complete [ 1610.720994] Lustre: 60060:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202068/real 1776202068] req@000000007c35ceda x1862482782066944/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202074 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1611.007148] Lustre: server umount lustre-MDT0000 complete [ 1622.421882] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1625.485385] Key type lgssc unregistered [ 1625.804732] LNet: 60664:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.817677] LNet: Removed LNI 192.168.204.148@tcp [ 1626.708983] Key type .llcrypt unregistered [ 1626.711376] Key type ._llcrypt unregistered [ 1657.627677] alg: No test for adler32 (adler32-zlib) [ 1658.383578] Key type ._llcrypt registered [ 1658.385446] Key type .llcrypt registered [ 1658.458260] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1659.629162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1660.428370] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 1660.831932] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1660.838350] LNet: Accept secure, port 988 [ 1662.554371] Key type lgssc registered [ 1664.491684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1674.466511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.840169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.068439] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1687.945747] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1694.747317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1695.102698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.861254] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1705.324818] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1705.421275] Lustre: 63628:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1705.433025] Lustre: 63628:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1705.442514] Lustre: 63628:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1705.451401] Lustre: 63628:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1705.461482] Lustre: 63628:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1705.473385] Lustre: 63628:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1710.107479] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1716.181743] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1725.920517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.949959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.041899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.576131] Lustre: server umount lustre-OST0000 complete [ 1741.791182] Lustre: 64592:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776202199/real 1776202199] req@000000001135158d x1862482918381888/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776202205 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1741.950221] Lustre: server umount lustre-MDT0000 complete [ 1753.479738] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1756.134133] Key type lgssc unregistered [ 1756.400757] LNet: 65196:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.414080] LNet: Removed LNI 192.168.204.148@tcp [ 1757.339557] Key type .llcrypt unregistered [ 1757.346312] Key type ._llcrypt unregistered [ 1776.061862] alg: No test for adler32 (adler32-zlib) [ 1776.816494] Key type ._llcrypt registered [ 1776.818245] Key type .llcrypt registered [ 1776.932428] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 1795.560741] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1797.340734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1798.177642] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 1798.399912] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1798.403815] LNet: Accept secure, port 988 [ 1800.119216] Key type lgssc registered [ 1800.801880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1808.776819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1815.911463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1823.007621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1830.122896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1836.343926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1836.402965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.632809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1837.675484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1837.764684] Lustre: lustre-MDT0000: new disk, initializing [ 1837.866486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.896950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1841.358579] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1851.618904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.681586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.755856] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1851.802307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1851.807468] Lustre: Skipped 1 previous similar message [ 1851.872140] Lustre: lustre-MDT0001: new disk, initializing [ 1851.949775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1851.988580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1852.002707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1856.333491] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1866.978366] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1873.614463] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1880.346162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1880.463854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.474809] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1880.742656] Lustre: lustre-OST0000: new disk, initializing [ 1880.748062] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1880.842989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.693475] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 1890.370511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1890.390871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1895.265451] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1900.043564] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1900.273507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1904.795196] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1904.982644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1911.265193] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1911.272689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.276027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1911.284537] Lustre: Skipped 1 previous similar message [ 1911.297960] Lustre: Skipped 1 previous similar message [ 1912.878912] Lustre: server umount lustre-OST0000 complete [ 1918.433899] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1918.443670] LustreError: Skipped 1 previous similar message [ 1918.451529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.466323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.505440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.509145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.538582] Lustre: Skipped 1 previous similar message [ 1922.854369] Lustre: server umount lustre-MDT0000 complete [ 1926.631444] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.650360] LustreError: Skipped 1 previous similar message [ 1926.671033] LustreError: 70586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202390 with bad export cookie 17970490295266950914 [ 1926.678358] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.682933] LustreError: 70586:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1927.051417] Lustre: server umount lustre-MDT0001 complete [ 1936.286719] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 17:33:17 (1776202397) [ 1946.977246] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1960.764182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1992.967520] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2001.835473] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2011.658843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2018.096828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2023.735280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2031.402751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2039.628890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2039.666817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.854962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2039.883896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2039.983552] Lustre: lustre-MDT0000: new disk, initializing [ 2040.062892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.078309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2044.809568] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2056.961987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2057.064504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.237947] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2057.333503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2057.349586] Lustre: Skipped 1 previous similar message [ 2057.516247] Lustre: lustre-MDT0001: new disk, initializing [ 2057.756498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2057.834955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2057.856176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2062.522863] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2074.555479] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2082.094793] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2090.635577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2090.773436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.779638] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2091.173930] Lustre: lustre-OST0000: new disk, initializing [ 2091.178718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2091.331862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.498540] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2099.365509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2099.395401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2106.115847] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2112.341693] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2112.577120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2117.433503] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2117.754425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2120.169348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.170902] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2120.178093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2120.186442] Lustre: Skipped 1 previous similar message [ 2120.208086] LustreError: Skipped 1 previous similar message [ 2125.284918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2125.297847] Lustre: Skipped 2 previous similar messages [ 2126.343969] Lustre: server umount lustre-OST0000 complete [ 2130.401467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.413180] Lustre: Skipped 1 previous similar message [ 2135.521898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.524091] Lustre: Skipped 2 previous similar messages [ 2144.223132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2144.320235] Lustre: server umount lustre-MDT0000 complete [ 2145.762952] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.789664] LustreError: Skipped 1 previous similar message [ 2146.797417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.821828] LustreError: Skipped 1 previous similar message [ 2148.048458] LustreError: 76792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202611 with bad export cookie 17970490295266951768 [ 2148.051680] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2148.059938] LustreError: 76792:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2157.510654] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 17:36:58 (1776202618) [ 2164.313398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.937060] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.073570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.343206] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.373605] LustreError: Skipped 1 previous similar message [ 2170.426788] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2179.754182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.330411] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2194.020969] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2199.892568] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2206.260731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2206.529628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.537821] Lustre: Skipped 1 previous similar message [ 2210.502958] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2217.656333] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2220.990867] Lustre: 84098:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 3d60c824-2383-476e-804a-c2c709880b5a at adminstrative request [ 2222.051606] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.065392] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2227.172792] Lustre: MGS: Client 3d60c824-2383-476e-804a-c2c709880b5a (at 0@lo) reconnecting [ 2227.185276] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0xf96401b638fdc5a7 to 0xf96401b638fdc84e [ 2227.204484] Lustre: MGC192.168.204.148@tcp: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 2258.506848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2258.578047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.585944] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2258.716630] Lustre: lustre-OST0001: new disk, initializing [ 2258.721767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2262.959185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2268.292467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2268.299226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2272.750220] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2278.880257] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.884901] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.901269] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2278.920414] Lustre: Skipped 1 previous similar message [ 2278.932217] Lustre: Skipped 3 previous similar messages [ 2281.735350] Lustre: server umount lustre-OST0001 complete [ 2281.738669] Lustre: Skipped 1 previous similar message [ 2284.007318] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.023309] LustreError: Skipped 2 previous similar messages [ 2288.608514] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.636154] LustreError: Skipped 1 previous similar message [ 2297.832328] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2297.847943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.859154] Lustre: Skipped 2 previous similar messages [ 2302.295702] Lustre: server umount lustre-MDT0000 complete [ 2302.301355] Lustre: Skipped 1 previous similar message [ 2304.486469] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.517061] LustreError: Skipped 3 previous similar messages [ 2306.164613] LustreError: 84496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202770 with bad export cookie 17970490295266953294 [ 2306.165709] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.178902] LustreError: 84496:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2315.854781] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 17:39:37 (1776202777) [ 2317.851453] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2319.790854] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 17:39:41 (1776202781) [ 2321.849621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2323.889713] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 17:39:45 (1776202785) [ 2325.754783] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2327.353566] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 17:39:49 (1776202789) [ 2328.822650] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2333.723480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2334.666431] loop: module loaded [ 2334.672811] loop0: detected capacity change from 0 to 4294967296 [ 2334.709210] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2336.246041] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2340.063416] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2344.271250] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2354.238350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.273401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2361.998906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2365.775646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2372.593834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2372.695305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2372.708719] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2372.728196] LustreError: 88550:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2372.736465] LustreError: 88550:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2372.744364] LustreError: 88550:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2372.760316] LustreError: 88550:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2389.418412] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2398.780171] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2408.979912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2415.446661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2421.064459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2427.173629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.226382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2434.272338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.482599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2434.518327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2434.640101] Lustre: lustre-MDT0000: new disk, initializing [ 2434.767705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.771218] Lustre: Skipped 1 previous similar message [ 2434.783743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2439.468927] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2450.083743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.149970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.244466] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2450.523750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2454.812385] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2464.161857] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2469.370134] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2475.817207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2475.885440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.892150] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2480.755371] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2481.753186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2491.190418] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2496.570888] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2496.879936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2502.108833] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2502.410338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2507.747428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.748284] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.758092] Lustre: Skipped 1 previous similar message [ 2507.766541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.784584] Lustre: Skipped 8 previous similar messages [ 2510.868859] Lustre: server umount lustre-OST0000 complete [ 2510.878184] Lustre: Skipped 1 previous similar message [ 2529.247552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2532.640718] LustreError: 91407:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776202996 with bad export cookie 17970490295266953875 [ 2532.650947] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.656372] LustreError: 91407:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2541.998500] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 17:43:23 (1776203003) [ 2548.349106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.785918] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.334059] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2563.649888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.140557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2564.144645] Lustre: Skipped 3 previous similar messages [ 2569.868664] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2578.107851] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2584.616641] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2591.379479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.599227] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2605.209624] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2611.909371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2611.978702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.983969] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2612.076826] Lustre: lustre-OST0001: new disk, initializing [ 2612.079965] Lustre: Skipped 2 previous similar messages [ 2612.083941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2612.088400] Lustre: Skipped 3 previous similar messages [ 2617.199119] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2620.504512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2620.511067] Lustre: Skipped 2 previous similar messages [ 2620.516722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2627.990655] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2650.078485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2675.632432] Lustre: Failing over lustre-MDT0000 [ 2676.098875] Lustre: server umount lustre-MDT0000 complete [ 2676.103601] Lustre: Skipped 2 previous similar messages [ 2676.705734] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2676.716026] LustreError: Skipped 1 previous similar message [ 2676.722706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.744832] Lustre: Skipped 3 previous similar messages [ 2676.751800] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.783771] LustreError: Skipped 2 previous similar messages [ 2684.895172] Lustre: 66725:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203141/real 1776203141] req@00000000e419e94a x1862483063206912/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776203148 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2684.924957] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.120683] LDISKFS-fs (dm-0): recovery complete [ 2704.124463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.675092] Lustre: Evicted from MGS (at 192.168.204.148@tcp) after server handle changed from 0xf96401b638fdcde9 to 0xf96401b638fe5bcc [ 2717.695760] Lustre: MGC192.168.204.148@tcp: Connection restored to (at 0@lo) [ 2718.026451] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2721.799133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2721.800671] Lustre: 95961:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2723.315055] Lustre: 95967:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2723.331737] Lustre: 95967:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2723.365338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2723.462110] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2723.792583] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2724.131257] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2724.143747] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2724.289973] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2724.480631] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2724.501014] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2724.514150] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2724.522209] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2725.167849] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2725.176388] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2725.496890] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2725.516574] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2726.798527] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2726.813971] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 2727.462667] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2727.471859] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2727.793478] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2727.799737] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2730.943926] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2730.971392] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2731.616732] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2731.635775] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2731.959132] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2731.968575] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2739.216935] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2739.232529] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 25 previous similar messages [ 2739.871997] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2739.890307] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2740.223134] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2740.246149] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 25 previous similar messages [ 2755.463549] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2755.496122] Lustre: 101148:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 2756.139627] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2756.159726] LustreError: 101148:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2756.479227] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2756.486943] LustreError: 101148:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2781.024848] Lustre: 101148:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2781.041167] Lustre: 101148:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2781.059916] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2781.069772] LustreError: 101148:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2781.083274] Lustre: 101148:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@00000000da20d37a x1862483043185664/t0(8589934777) o101->5eaa610b-75fa-447b-8e26-04e6133fb7b5@192.168.204.48@tcp:65/0 lens 592/0 e 0 to 0 dl 1776203250 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2781.110814] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2781.113696] Lustre: Skipped 2 previous similar messages [ 2781.116237] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2781.190729] Lustre: lustre-OST0000: deleting orphan objects from 0x0:79 to 0x0:577 [ 2781.191765] Lustre: lustre-OST0001: deleting orphan objects from 0x0:47 to 0x0:545 [ 2794.987217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2794.995714] Lustre: Skipped 10 previous similar messages [ 2808.287436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2808.340917] Lustre: server umount lustre-OST0000 complete [ 2810.344285] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.355715] LustreError: Skipped 69 previous similar messages [ 2812.896520] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2812.911588] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.947379] Lustre: Skipped 5 previous similar messages [ 2827.231421] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2847.199364] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2851.788377] LustreError: 98873:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203315 with bad export cookie 17970490295266991052 [ 2851.789957] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2851.800595] LustreError: 98873:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2861.387706] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 17:48:42 (1776203322) [ 2870.824302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.833673] Lustre: *** cfs_fail_loc=197, val=0*** [ 2888.543737] LustreError: 103181:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 1: rc = -5 [ 2899.808381] LustreError: 103181:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 2906.010896] Lustre: *** cfs_fail_loc=197, val=0*** [ 2906.019589] Lustre: Skipped 65 previous similar messages [ 2929.633647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.649033] Lustre: Skipped 3 previous similar messages [ 2935.799199] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 2944.569491] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2956.321590] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 17:50:18 (1776203418) [ 2980.145678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.336391] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3000.909258] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3005.867993] LustreError: 104839:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3005.883138] LustreError: 104839:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3005.890701] LustreError: 104839:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 3011.552608] Lustre: 105531:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776203469/real 1776203469] req@0000000022549bb2 x1862483063246336/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776203475 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3030.305132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3052.202882] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3063.199408] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3073.740497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.579540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3088.621184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3095.559216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3103.859164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3103.973719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.394676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3104.463324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3104.558665] Lustre: lustre-MDT0000: new disk, initializing [ 3104.699652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3109.894503] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3121.549612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3121.618648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.727792] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3121.808326] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3121.818779] Lustre: Skipped 17 previous similar messages [ 3122.046853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3127.353310] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3138.508938] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.659771] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3152.942297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3153.060311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.067233] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3157.967768] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3167.859729] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3172.670771] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3172.947756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3178.288356] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3178.688807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3184.098840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.103177] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3184.124340] Lustre: Skipped 3 previous similar messages [ 3184.145930] LustreError: Skipped 1 previous similar message [ 3195.359537] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3195.558890] Lustre: server umount lustre-OST0000 complete [ 3195.561187] Lustre: Skipped 5 previous similar messages [ 3209.696939] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.720547] LustreError: Skipped 15 previous similar messages [ 3211.625169] LustreError: 111256:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203675 with bad export cookie 17970490295267001692 [ 3211.632766] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3211.638433] LustreError: 111256:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3223.118693] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 17:54:44 (1776203684) [ 3234.340984] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3237.269774] Key type lgssc unregistered [ 3237.646544] LNet: 113853:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.668254] LNet: Removed LNI 192.168.204.148@tcp [ 3238.613779] Key type .llcrypt unregistered [ 3238.621481] Key type ._llcrypt unregistered [ 3261.427588] alg: No test for adler32 (adler32-zlib) [ 3262.189279] Key type ._llcrypt registered [ 3262.197349] Key type .llcrypt registered [ 3262.459495] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3280.650845] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3282.225227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3283.212129] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3283.529918] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3283.536309] LNet: Accept secure, port 988 [ 3285.231292] Key type lgssc registered [ 3286.852102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3298.172908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3305.880790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3312.778444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3320.074960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3326.927343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3335.062248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3335.143935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.438487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3336.478357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3336.585803] Lustre: lustre-MDT0000: new disk, initializing [ 3336.714633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.746617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3341.431447] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3353.256832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3353.331762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3353.341621] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3353.667740] Lustre: lustre-OST0000: new disk, initializing [ 3353.677678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3353.682666] Lustre: Skipped 1 previous similar message [ 3353.769170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.798324] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3370.487866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3370.592469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.598834] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3370.751668] Lustre: lustre-OST0001: new disk, initializing [ 3370.759475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3370.836042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3374.854496] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3384.300821] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.312852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3384.322947] Lustre: Skipped 1 previous similar message [ 3389.417187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.424041] Lustre: Skipped 2 previous similar messages [ 3389.773887] Lustre: server umount lustre-MDT0000 complete [ 3396.261658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3401.175751] LustreError: 117366:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776203865 with bad export cookie 7846238658010192092 [ 3401.183710] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3401.206399] LustreError: 117366:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3401.251406] Lustre: server umount lustre-OST0000 complete [ 3405.082683] Lustre: server umount lustre-OST0001 complete [ 3413.778740] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 17:57:55 (1776203875) [ 3425.493839] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3435.614622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3436.699747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3442.950771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3444.205696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3450.980743] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 17:58:32 (1776203912) [ 3464.968422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3467.091824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3468.970161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3471.056764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3472.017510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.945160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3492.510333] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3502.222813] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3512.825979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3519.619182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3526.920991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3534.147861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.868202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.960884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3542.194648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3542.243222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3542.327807] Lustre: lustre-MDT0000: new disk, initializing [ 3542.441227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.470115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3547.576978] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3559.459907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3559.563069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.688032] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3559.746257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3559.760328] Lustre: Skipped 1 previous similar message [ 3559.852919] Lustre: lustre-MDT0001: new disk, initializing [ 3559.948347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.981890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3560.006235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3565.148470] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3576.053452] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3582.125607] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3589.511356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3589.596656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3589.616123] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3590.140159] Lustre: lustre-OST0000: new disk, initializing [ 3590.149267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3590.277982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.887254] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3597.934034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3597.948649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3605.282610] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3611.544863] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3611.808629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3618.028903] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3618.266370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3623.908857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.914969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3623.918324] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3623.926273] Lustre: Skipped 1 previous similar message [ 3623.946722] Lustre: Skipped 1 previous similar message [ 3626.889279] Lustre: server umount lustre-OST0000 complete [ 3631.591102] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3631.603661] LustreError: Skipped 1 previous similar message [ 3631.612750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.635376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.144839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.772411] Lustre: server umount lustre-MDT0000 complete [ 3639.271430] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.297585] LustreError: Skipped 1 previous similar message [ 3640.290271] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.317405] LustreError: Skipped 1 previous similar message [ 3642.176706] LustreError: 125911:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204106 with bad export cookie 7846238658010193289 [ 3642.181698] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.190547] LustreError: 125911:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3654.254544] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 18:01:55 (1776204115) [ 3662.928362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.463917] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.527185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.830375] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3668.969824] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.085621] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.185442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.290825] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3692.033530] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3698.466511] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3705.674311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3705.988927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3705.997836] Lustre: Skipped 1 previous similar message [ 3711.301189] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3720.464655] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3725.759335] Lustre: *** cfs_fail_loc=159, val=0*** [ 3732.044341] Lustre: lustre-MDT0000: Client 762f64bb-1744-4cc6-8e0d-8ada439c31bd (at 192.168.204.48@tcp) reconnecting [ 3733.198726] Lustre: *** cfs_fail_loc=159, val=0*** [ 3733.203209] Lustre: Skipped 7 previous similar messages [ 3740.765018] Lustre: lustre-MDT0000: Client 762f64bb-1744-4cc6-8e0d-8ada439c31bd (at 192.168.204.48@tcp) reconnecting [ 3747.911446] Lustre: server umount lustre-OST0000 complete [ 3747.913731] Lustre: Skipped 1 previous similar message [ 3754.480826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.486412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3754.506116] Lustre: Skipped 1 previous similar message [ 3754.516287] Lustre: Skipped 2 previous similar messages [ 3757.087742] Lustre: 115379:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204213/real 1776204213] req@00000000b60fde3f x1862484620264256/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776204220 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3758.175878] Lustre: 115378:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204214/real 1776204214] req@000000006f752ebd x1862484620264448/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776204221 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3758.221831] Lustre: 115378:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3761.632206] Lustre: 115380:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204218/real 1776204218] req@00000000bb21fedc x1862484620265280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776204225 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3762.150040] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.166706] LustreError: Skipped 2 previous similar messages [ 3763.907429] LustreError: 131272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204227 with bad export cookie 7846238658010194143 [ 3763.917659] LustreError: 131272:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3763.921714] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3764.312297] Lustre: server umount lustre-MDT0001 complete [ 3764.316247] Lustre: Skipped 1 previous similar message [ 3776.667866] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3779.517168] Key type lgssc unregistered [ 3779.909851] LNet: 134338:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3779.921586] LNet: Removed LNI 192.168.204.148@tcp [ 3780.810398] Key type .llcrypt unregistered [ 3780.813757] Key type ._llcrypt unregistered [ 3805.623414] alg: No test for adler32 (adler32-zlib) [ 3806.379533] Key type ._llcrypt registered [ 3806.385905] Key type .llcrypt registered [ 3806.516685] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3808.001189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3808.663585] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 3808.996021] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3809.010334] LNet: Accept secure, port 988 [ 3810.711127] Key type lgssc registered [ 3811.991717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3820.745888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.248185] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.311807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.136224] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3827.692037] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3832.808197] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.185966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.546258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.398428] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3849.932532] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3856.834122] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3863.981446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3864.351643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.223912] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 3872.557827] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3877.424044] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3887.965545] Lustre: *** cfs_fail_loc=159, val=0*** [ 3895.407837] Lustre: lustre-MDT0000: Client 25472faa-1bcb-4a89-a0b6-9f4f30075cf2 (at 192.168.204.48@tcp) reconnecting [ 3898.088308] Lustre: *** cfs_fail_loc=159, val=0*** [ 3905.593789] Lustre: lustre-MDT0001: Client 25472faa-1bcb-4a89-a0b6-9f4f30075cf2 (at 192.168.204.48@tcp) reconnecting [ 3907.144174] Lustre: *** cfs_fail_loc=159, val=0*** [ 3907.146336] Lustre: Skipped 3 previous similar messages [ 3914.358615] Lustre: lustre-MDT0001: Client 25472faa-1bcb-4a89-a0b6-9f4f30075cf2 (at 192.168.204.48@tcp) reconnecting [ 3924.206143] Lustre: *** cfs_fail_loc=159, val=0*** [ 3924.208165] Lustre: Skipped 2 previous similar messages [ 3931.697656] Lustre: lustre-MDT0000: Client b34372a4-cb2f-49a6-9dbd-3a8bbe907fb1 (at 192.168.204.48@tcp) reconnecting [ 3932.841275] Lustre: *** cfs_fail_loc=159, val=0*** [ 3932.844561] Lustre: Skipped 14 previous similar messages [ 3940.432986] Lustre: lustre-MDT0000: Client b34372a4-cb2f-49a6-9dbd-3a8bbe907fb1 (at 192.168.204.48@tcp) reconnecting [ 3948.010325] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3948.016990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.018989] LustreError: Skipped 1 previous similar message [ 3948.044556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3948.056834] Lustre: Skipped 1 previous similar message [ 3949.548624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3949.559444] Lustre: Skipped 1 previous similar message [ 3953.727935] Lustre: server umount lustre-OST0000 complete [ 3959.268404] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.274122] LustreError: Skipped 3 previous similar messages [ 3959.286165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.304379] Lustre: Skipped 1 previous similar message [ 3959.311703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.949275] Lustre: server umount lustre-MDT0000 complete [ 3964.323758] LustreError: 135804:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204428 with bad export cookie 4411396368658841370 [ 3964.337728] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.832872] Lustre: server umount lustre-MDT0001 complete [ 3978.102537] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3981.562287] Key type lgssc unregistered [ 3981.960927] LNet: 140211:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3981.968991] LNet: Removed LNI 192.168.204.148@tcp [ 3982.804200] Key type .llcrypt unregistered [ 3982.806680] Key type ._llcrypt unregistered [ 4010.516877] alg: No test for adler32 (adler32-zlib) [ 4011.286490] Key type ._llcrypt registered [ 4011.288890] Key type .llcrypt registered [ 4011.413746] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4012.517115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4013.203893] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4013.487505] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4013.491474] LNet: Accept secure, port 988 [ 4015.175165] Key type lgssc registered [ 4016.245948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4024.621126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.190264] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.259884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.459711] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.700527] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4036.595625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.733958] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.930531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.462771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.276730] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4055.933059] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4062.274559] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4069.883438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4070.290364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.408978] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 4075.918762] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4076.485549] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 4085.428597] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4097.765239] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 4098.604869] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 4102.148277] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 4112.357588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.364992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.371307] Lustre: Skipped 1 previous similar message [ 4112.391687] Lustre: Skipped 1 previous similar message [ 4114.821688] Lustre: server umount lustre-OST0000 complete [ 4119.009315] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4119.015926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.031441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4122.593730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4122.596405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4124.817444] Lustre: server umount lustre-MDT0000 complete [ 4127.714517] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.731625] LustreError: Skipped 1 previous similar message [ 4128.340974] LustreError: 141681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204592 with bad export cookie 11562110206523565051 [ 4128.349132] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4128.350155] LustreError: 141681:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4128.606702] Lustre: server umount lustre-MDT0001 complete [ 4141.294633] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4145.538642] Key type lgssc unregistered [ 4146.044449] LNet: 145761:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4146.053779] LNet: Removed LNI 192.168.204.148@tcp [ 4147.020892] Key type .llcrypt unregistered [ 4147.022552] Key type ._llcrypt unregistered [ 4175.170219] alg: No test for adler32 (adler32-zlib) [ 4175.928821] Key type ._llcrypt registered [ 4175.932577] Key type .llcrypt registered [ 4176.074183] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4177.025736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4177.994725] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4178.265582] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4178.279635] LNet: Accept secure, port 988 [ 4180.015328] Key type lgssc registered [ 4181.174628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4188.590201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.054828] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.150462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.146079] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.935891] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4199.406672] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.519178] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.244718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.866533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4211.806512] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4220.447824] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4226.591473] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4234.417211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.951566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.018737] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 4240.128324] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 4240.390049] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4249.503558] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4255.057602] Lustre: *** cfs_fail_loc=159, val=0*** [ 4262.494122] Lustre: lustre-MDT0000: Client a699b7a4-d38a-4f63-a7d8-9118a12fb452 (at 192.168.204.48@tcp) reconnecting [ 4268.009427] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4268.016825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.027645] LustreError: Skipped 1 previous similar message [ 4268.076320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.589835] Lustre: server umount lustre-OST0000 complete [ 4276.195605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.208517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4276.210316] Lustre: Skipped 1 previous similar message [ 4276.236492] Lustre: Skipped 1 previous similar message [ 4277.215472] Lustre: 146559:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204734/real 1776204734] req@00000000e091cd78 x1862485558707392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776204741 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4277.257504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.288292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4277.304108] Lustre: Skipped 1 previous similar message [ 4278.303246] Lustre: 146558:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204735/real 1776204735] req@00000000f2cabba2 x1862485558707584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1776204742 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4280.716327] Lustre: server umount lustre-MDT0000 complete [ 4282.338660] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.365328] LustreError: Skipped 1 previous similar message [ 4283.359562] Lustre: 146559:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776204740/real 1776204740] req@00000000c9e79c31 x1862485558708288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776204747 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4285.616808] LustreError: 148037:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204749 with bad export cookie 2063797351134272726 [ 4285.619558] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4285.626056] LustreError: 148037:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4285.939255] Lustre: server umount lustre-MDT0001 complete [ 4299.348549] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4302.706823] Key type lgssc unregistered [ 4303.033669] LNet: 151009:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4303.047381] LNet: Removed LNI 192.168.204.148@tcp [ 4304.221351] Key type .llcrypt unregistered [ 4304.228512] Key type ._llcrypt unregistered [ 4331.098673] alg: No test for adler32 (adler32-zlib) [ 4331.855778] Key type ._llcrypt registered [ 4331.860461] Key type .llcrypt registered [ 4332.037628] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4333.400340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4333.858724] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4333.972931] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4333.976511] LNet: Accept secure, port 988 [ 4335.663196] Key type lgssc registered [ 4336.832527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4345.929542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4347.541966] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.651412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4352.999815] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.192110] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4358.113338] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.213673] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.961589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.633470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4368.778510] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4378.239738] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4385.114907] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4392.621484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4392.917685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.972124] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 4397.916118] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4398.074162] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 4407.177805] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4427.018225] Lustre: lustre-OST0000: evicting 07aaf7cc-a121-4435-8ef9-e20b6a8183f7 (at 192.168.204.48@tcp) by administrative request [ 4427.028793] Lustre: Skipped 1 previous similar message [ 4459.490730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.492494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4459.518777] Lustre: Skipped 1 previous similar message [ 4459.544207] Lustre: Skipped 1 previous similar message [ 4461.874766] Lustre: server umount lustre-OST0000 complete [ 4469.731742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.751050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4473.003957] Lustre: server umount lustre-MDT0000 complete [ 4474.849298] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.878284] LustreError: Skipped 1 previous similar message [ 4478.138346] LustreError: 152478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776204942 with bad export cookie 5013660368150823999 [ 4478.142602] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.145882] LustreError: 152478:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4478.743675] Lustre: server umount lustre-MDT0001 complete [ 4493.224645] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4497.001688] Key type lgssc unregistered [ 4497.440774] LNet: 156693:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4497.446298] LNet: Removed LNI 192.168.204.148@tcp [ 4498.519247] Key type .llcrypt unregistered [ 4498.528815] Key type ._llcrypt unregistered [ 4589.078763] alg: No test for adler32 (adler32-zlib) [ 4589.845441] Key type ._llcrypt registered [ 4589.856709] Key type .llcrypt registered [ 4590.058869] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4592.031289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4592.817197] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4593.153909] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4593.169553] LNet: Accept secure, port 988 [ 4594.890823] Key type lgssc registered [ 4596.508726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4606.149987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.705027] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4607.783047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.906590] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4613.096455] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.209872] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.847568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4622.223835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4627.144400] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4635.618327] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4641.948624] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4649.317343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4649.604877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.735029] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 4654.249086] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4658.889675] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 4664.157294] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4674.529303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.530344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.530445] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4674.548634] Lustre: Skipped 1 previous similar message [ 4674.561762] Lustre: Skipped 1 previous similar message [ 4679.662252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.442779] Lustre: server umount lustre-OST0000 complete [ 4688.864915] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4688.880043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.897778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.906205] Lustre: Skipped 1 previous similar message [ 4689.906776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.443640] Lustre: server umount lustre-MDT0000 complete [ 4695.014389] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.033044] LustreError: Skipped 1 previous similar message [ 4696.732749] LustreError: 159239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205160 with bad export cookie 8087702683769303503 [ 4696.744218] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4696.749228] LustreError: 159239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4697.425711] Lustre: server umount lustre-MDT0001 complete [ 4713.685594] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4717.248968] Key type lgssc unregistered [ 4717.565865] LNet: 162926:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4717.591081] LNet: Removed LNI 192.168.204.148@tcp [ 4718.655363] Key type .llcrypt unregistered [ 4718.658584] Key type ._llcrypt unregistered [ 4745.103393] alg: No test for adler32 (adler32-zlib) [ 4745.861772] Key type ._llcrypt registered [ 4745.867561] Key type .llcrypt registered [ 4746.013394] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4747.290250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4748.314297] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 4748.696483] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4748.699783] LNet: Accept secure, port 988 [ 4750.492506] Key type lgssc registered [ 4751.686789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4760.135048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4768.398480] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 18:20:29 (1776205229) [ 4788.596559] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4800.080120] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4812.968817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4820.863953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4828.926637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4836.936566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4844.211314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4844.260657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.650395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4845.772052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4845.825298] Lustre: lustre-MDT0000: new disk, initializing [ 4845.922953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.954364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4852.032730] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4865.334976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4865.442701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.615340] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4865.709298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4865.725710] Lustre: Skipped 1 previous similar message [ 4865.909456] Lustre: lustre-MDT0001: new disk, initializing [ 4866.078883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4866.147573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4866.174435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4870.987870] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4882.909330] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4890.048751] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4898.437690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4898.520197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4898.528802] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4898.755415] Lustre: lustre-OST0000: new disk, initializing [ 4898.764158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4898.887318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4904.853124] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 4907.586181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4907.594169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4916.292931] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.485729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4928.500891] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4928.505761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4928.529927] Lustre: Skipped 1 previous similar message [ 4933.602351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.614092] Lustre: Skipped 1 previous similar message [ 4933.903356] Lustre: server umount lustre-OST0000 complete [ 4938.720291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.746185] Lustre: Skipped 1 previous similar message [ 4938.760887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4938.776844] Lustre: Skipped 1 previous similar message [ 4943.840905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.856094] Lustre: Skipped 1 previous similar message [ 4944.890635] Lustre: server umount lustre-MDT0000 complete [ 4948.961709] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.976921] LustreError: Skipped 1 previous similar message [ 4949.224567] LustreError: 168209:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205413 with bad export cookie 621000637654848190 [ 4949.251376] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.779317] Lustre: server umount lustre-MDT0001 complete [ 4962.917727] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4966.275504] Key type lgssc unregistered [ 4966.682959] LNet: 171361:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4966.708098] LNet: Removed LNI 192.168.204.148@tcp [ 4967.612780] Key type .llcrypt unregistered [ 4967.617426] Key type ._llcrypt unregistered [ 4997.419466] alg: No test for adler32 (adler32-zlib) [ 4998.177509] Key type ._llcrypt registered [ 4998.181531] Key type .llcrypt registered [ 4998.375994] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5000.335268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5001.503491] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 5001.899920] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5001.910808] LNet: Accept secure, port 988 [ 5003.783182] Key type lgssc registered [ 5005.084194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5015.791563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.614891] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.718169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.221671] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.510700] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5028.325607] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.452231] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.926089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.547885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5041.691747] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5051.248404] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5059.656771] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5068.774896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5069.151898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.503159] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5084.979755] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5093.790308] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 5101.078347] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5101.098907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.129268] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5101.140791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.149197] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 5101.152051] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 5102.599288] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5102.618226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.636979] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5102.642710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.656856] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 5102.661925] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 5104.425975] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5104.463537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.483677] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5104.494969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.522135] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 5104.532453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:129 [ 5106.030053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.070507] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:161 [ 5107.396071] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5107.411804] Lustre: Skipped 1 previous similar message [ 5107.456741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5107.470528] Lustre: Skipped 1 previous similar message [ 5107.482503] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5107.491638] Lustre: Skipped 1 previous similar message [ 5107.502568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.534423] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 5107.542480] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:193 [ 5107.562615] Lustre: Skipped 1 previous similar message [ 5108.987203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.022457] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:225 [ 5110.122462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.142808] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:257 [ 5111.341157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.359752] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:289 [ 5112.391945] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5112.396251] Lustre: Skipped 3 previous similar messages [ 5112.411571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.423234] Lustre: Skipped 3 previous similar messages [ 5112.425784] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5112.431964] Lustre: Skipped 3 previous similar messages [ 5112.437845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.449641] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 5112.453059] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:321 [ 5112.464952] Lustre: Skipped 3 previous similar messages [ 5113.752851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.772587] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:353 [ 5124.064267] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5124.074109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.090019] Lustre: Skipped 1 previous similar message [ 5124.100427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.697075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5129.702413] Lustre: Skipped 2 previous similar messages [ 5130.177721] Lustre: server umount lustre-OST0000 complete [ 5134.828919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5134.831854] Lustre: Skipped 1 previous similar message [ 5139.936380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.951704] Lustre: Skipped 1 previous similar message [ 5140.961760] Lustre: server umount lustre-MDT0000 complete [ 5145.059488] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5145.075099] LustreError: Skipped 1 previous similar message [ 5145.291027] LustreError: 174801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205609 with bad export cookie 14520814186278384369 [ 5145.292100] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5145.303989] LustreError: 174801:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5145.917050] Lustre: server umount lustre-MDT0001 complete [ 5161.631207] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5165.464911] Key type lgssc unregistered [ 5165.842552] LNet: 177331:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5165.862168] LNet: Removed LNI 192.168.204.148@tcp [ 5167.083613] Key type .llcrypt unregistered [ 5167.091067] Key type ._llcrypt unregistered [ 5195.362645] alg: No test for adler32 (adler32-zlib) [ 5196.130831] Key type ._llcrypt registered [ 5196.138573] Key type .llcrypt registered [ 5196.275214] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5197.553897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5198.562650] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 5198.987047] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5198.993604] LNet: Accept secure, port 988 [ 5200.880251] Key type lgssc registered [ 5202.330531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5211.380145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.053949] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.137940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.279625] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.703388] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5223.396145] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.521235] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.481797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.973565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5235.977222] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5245.110749] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5252.508647] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.402535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5260.921514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.011575] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:385 [ 5266.696733] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5276.892915] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5285.218558] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5287.782598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.793761] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5287.800481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.806531] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.148@tcp (at 0@lo) [ 5287.807352] LustreError: 178837:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 5289.965820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.966534] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5289.993656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.008658] Lustre: Skipped 1 previous similar message [ 5294.119749] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 5295.772735] Lustre: server umount lustre-OST0000 complete [ 5307.903600] LustreError: 178837:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 5307.930021] LustreError: 178837:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 5331.960646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5332.305854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.460165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.480774] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.492194] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 5337.501335] Lustre: lustre-OST0000: deleting orphan objects from 0x0:113 to 0x0:417 [ 5337.514857] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 5338.034446] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5346.933780] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5349.245465] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5355.875423] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e0bc7902800.ost_server_uuid 40 [ 5358.049231] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e0bc7902800.ost_server_uuid in FULL state after 0 sec [ 5368.289279] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.292764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.294792] LustreError: Skipped 1 previous similar message [ 5368.296835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.300868] Lustre: Skipped 2 previous similar messages [ 5368.304334] Lustre: Skipped 2 previous similar messages [ 5369.657491] Lustre: server umount lustre-OST0000 complete [ 5378.528612] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.544696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.558407] Lustre: Skipped 1 previous similar message [ 5378.569371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.576786] Lustre: Skipped 2 previous similar messages [ 5383.648495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.659737] Lustre: Skipped 1 previous similar message [ 5388.255870] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5388.392060] Lustre: server umount lustre-MDT0000 complete [ 5389.793745] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.821513] LustreError: Skipped 1 previous similar message [ 5392.763416] LustreError: 178798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776205856 with bad export cookie 10561749871422105093 [ 5392.769872] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5392.779485] LustreError: 178798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5393.280384] Lustre: server umount lustre-MDT0001 complete [ 5408.363388] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5412.205691] Key type lgssc unregistered [ 5412.614067] LNet: 184115:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5412.640545] LNet: Removed LNI 192.168.204.148@tcp [ 5413.805826] Key type .llcrypt unregistered [ 5413.810682] Key type ._llcrypt unregistered [ 5449.413309] alg: No test for adler32 (adler32-zlib) [ 5450.170897] Key type ._llcrypt registered [ 5450.173614] Key type .llcrypt registered [ 5450.296671] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5451.173318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5451.666181] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 5451.898776] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5451.907602] LNet: Accept secure, port 988 [ 5453.543181] Key type lgssc registered [ 5454.786812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5466.030924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.856566] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.908807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5473.250069] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5473.866898] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5478.371924] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.491660] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.466809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.861878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5490.184193] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5500.262371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5500.737697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.810345] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5517.142564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5517.278531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5517.287368] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5517.517514] Lustre: lustre-OST0001: new disk, initializing [ 5517.529524] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5517.636433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5518.717162] Lustre: lustre-OST0000: deleting orphan objects from 0x0:419 to 0x0:449 [ 5524.185516] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5525.167368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5525.183908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5536.438591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5544.474737] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5566.440942] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.444544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5566.456380] Lustre: Skipped 3 previous similar messages [ 5566.474287] Lustre: Skipped 3 previous similar messages [ 5569.581290] Lustre: server umount lustre-MDT0000 complete [ 5571.554924] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.577877] LustreError: Skipped 3 previous similar messages [ 5574.375376] LustreError: 188246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206038 with bad export cookie 5721475271822848175 [ 5574.380388] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5574.422937] LustreError: 188246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5574.898133] Lustre: server umount lustre-MDT0001 complete [ 5583.775317] Lustre: 185178:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206040/real 1776206040] req@00000000949ea42d x1862486893574336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206047 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 5583.795093] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.351158] Lustre: 189704:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206042/real 1776206042] req@000000008b0c784b x1862486893574464/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206048 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5584.393768] Lustre: server umount lustre-OST0000 complete [ 5586.912268] Lustre: 185181:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206043/real 1776206043] req@0000000043203a10 x1862486893574528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206050 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 5589.422513] Lustre: server umount lustre-OST0001 complete [ 5593.991083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5594.172202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5595.734669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5595.848877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5597.401307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5597.577838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5598.834488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5598.954877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5615.528174] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5625.632760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.216982] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5626.310909] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.705667] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5642.090643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.875722] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5648.350736] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5658.153570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5663.698254] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5673.895835] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5673.918460] LustreError: Skipped 3 previous similar messages [ 5673.974831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5674.270090] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.274105] Lustre: Skipped 1 previous similar message [ 5677.819097] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:452 to 0x0:481 [ 5678.931393] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5687.800760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5691.927036] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5708.770830] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.779992] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5708.794083] Lustre: Skipped 2 previous similar messages [ 5708.813935] Lustre: Skipped 3 previous similar messages [ 5713.908145] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.910833] Lustre: Skipped 2 previous similar messages [ 5714.854628] Lustre: server umount mylustre-MDT0000 complete [ 5718.797953] LustreError: 191347:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206182 with bad export cookie 5721475271822850898 [ 5718.804086] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5718.820851] LustreError: 191347:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5726.112141] Lustre: 185181:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206182/real 1776206182] req@0000000046ab4575 x1862486893604736/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206189 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5726.148428] Lustre: 185181:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5726.156328] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.170885] Lustre: Skipped 1 previous similar message [ 5728.272629] Lustre: server umount mylustre-OST0000 complete [ 5728.275655] Lustre: Skipped 1 previous similar message [ 5730.271836] Lustre: 185179:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206187/real 1776206187] req@0000000050b698d0 x1862486893604992/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206194 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5730.308894] Lustre: 185179:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5736.632240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5736.765502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5738.186570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5738.303658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5739.730383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5739.825169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.981574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5741.076305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5759.187451] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5772.486481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.222266] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.341172] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.934840] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5787.769165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.864304] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5801.391624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5804.809595] Lustre: tfs-OST0000: deleting orphan objects from 0x0:452 to 0x0:513 [ 5807.136556] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5816.186752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5818.466816] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 5821.490640] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5830.546732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5834.829728] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5848.825241] Lustre: server umount tfs-MDT0000 complete [ 5848.831188] Lustre: Skipped 1 previous similar message [ 5849.584302] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5849.598578] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.616100] LustreError: 137-5: tfs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.644915] LustreError: Skipped 8 previous similar messages [ 5852.935134] LustreError: 199728:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206316 with bad export cookie 5721475271822852508 [ 5852.939445] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5852.944646] LustreError: 199728:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5863.391217] Lustre: 200381:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776206321/real 1776206321] req@000000007f26ca22 x1862486893632000/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776206327 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5863.436112] Lustre: 200381:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5864.351743] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.373382] Lustre: Skipped 3 previous similar messages [ 5871.650631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5871.755371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5872.862824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5872.991702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5874.269447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5874.381371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5875.529793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5875.642415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5894.192830] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5906.979892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.499452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.503364] Lustre: Skipped 3 previous similar messages [ 5913.113295] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5922.918268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.589627] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5938.249863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5943.375798] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5951.620054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5957.239537] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 5957.257214] Lustre: lustre-OST0000: deleting orphan objects from 0x0:452 to 0x0:545 [ 5958.826599] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 5969.786793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5974.184268] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5988.073481] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 18:40:49 (1776206449) [ 6005.216814] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6005.227709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.270183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6010.553188] Lustre: server umount lustre-MDT0000 complete [ 6010.555281] Lustre: Skipped 3 previous similar messages [ 6013.920334] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.930422] LustreError: Skipped 12 previous similar messages [ 6014.431413] LustreError: 202025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206478 with bad export cookie 5721475271822854139 [ 6014.436272] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.440766] LustreError: 202025:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6019.060102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6019.067725] Lustre: Skipped 5 previous similar messages [ 6035.769743] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6046.068385] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6058.080414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6066.755033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6074.903237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6083.320664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6090.642176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6090.753784] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6090.753784] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6090.753784] [ 6090.809246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6091.296611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6091.372325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6091.562575] Lustre: lustre-MDT0000: new disk, initializing [ 6091.702232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.719333] Lustre: Skipped 3 previous similar messages [ 6091.786045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6098.812942] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6111.888935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6111.983236] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 6111.983236] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6111.983236] [ 6112.033180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6112.197257] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6112.264707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6112.279262] Lustre: Skipped 1 previous similar message [ 6112.402656] Lustre: lustre-MDT0001: new disk, initializing [ 6112.582827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6112.595195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6117.944335] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6130.176668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6130.286368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6130.293450] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6130.634802] Lustre: lustre-OST0000: new disk, initializing [ 6130.639256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6132.330939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6132.350520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6136.065866] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6147.728183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6147.797278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.803681] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6147.921895] Lustre: lustre-OST0001: new disk, initializing [ 6147.932145] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6149.405322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6149.427935] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6153.910500] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6167.888906] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 18:43:49 (1776206629) [ 6184.603270] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 18:44:06 (1776206646) [ 6200.814542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.824963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.832240] Lustre: Skipped 6 previous similar messages [ 6200.840219] Lustre: Skipped 1 previous similar message [ 6205.711223] Lustre: server umount lustre-OST0000 complete [ 6205.715067] Lustre: Skipped 3 previous similar messages [ 6210.016182] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6221.283209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6221.297436] Lustre: Skipped 9 previous similar messages [ 6224.352734] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6228.703329] LustreError: 208309:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206692 with bad export cookie 5721475271822855658 [ 6228.704238] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.711025] LustreError: 208309:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6247.961652] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6268.793770] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6283.755289] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6286.357646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6296.089540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6305.451049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6313.065275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6320.405559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6329.224801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6329.313403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.723512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6329.805194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6329.927903] Lustre: lustre-MDT0000: new disk, initializing [ 6330.098540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6335.622753] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6348.098242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6348.167679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.306476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6348.321668] Lustre: Skipped 1 previous similar message [ 6348.478729] Lustre: lustre-MDT0001: new disk, initializing [ 6348.660986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6348.671192] Lustre: Skipped 4 previous similar messages [ 6348.726824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6348.739735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6354.580465] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6365.722691] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6372.518616] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6381.029695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.100075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.119708] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6381.580989] Lustre: lustre-OST0000: new disk, initializing [ 6381.597990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6383.388661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6383.408373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6387.506974] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6399.275556] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6409.207099] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.238445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.259682] Lustre: Skipped 4 previous similar messages [ 6409.268994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6409.270918] Lustre: Skipped 3 previous similar messages [ 6412.287820] Lustre: server umount lustre-OST0000 complete [ 6412.289722] Lustre: Skipped 3 previous similar messages [ 6424.544859] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6424.556599] LustreError: Skipped 18 previous similar messages [ 6427.923630] LustreError: 215527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776206891 with bad export cookie 5721475271822857086 [ 6427.925845] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6427.934517] LustreError: 215527:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6441.683659] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6444.823672] Key type lgssc unregistered [ 6445.080414] LNet: 219391:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6445.102509] LNet: Removed LNI 192.168.204.148@tcp [ 6445.998850] Key type .llcrypt unregistered [ 6446.003375] Key type ._llcrypt unregistered [ 6473.382656] alg: No test for adler32 (adler32-zlib) [ 6474.139775] Key type ._llcrypt registered [ 6474.148664] Key type .llcrypt registered [ 6474.295493] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6475.728430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6476.443469] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 6476.695289] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6476.702222] LNet: Accept secure, port 988 [ 6478.399143] Key type lgssc registered [ 6479.569995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6488.634202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.257329] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6490.350305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.716958] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.018956] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6500.833058] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.957146] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6506.627103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6506.977178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6511.656967] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6520.481164] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6526.575548] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6533.229317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.576465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6538.175564] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6540.789737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 6546.353824] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6547.469507] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 6551.015851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6551.019686] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6551.049351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6551.066523] Lustre: Skipped 1 previous similar message [ 6555.778095] Lustre: server umount lustre-OST0000 complete [ 6561.249058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.253173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6561.260222] Lustre: Skipped 2 previous similar messages [ 6561.275387] Lustre: Skipped 1 previous similar message [ 6565.593046] Lustre: server umount lustre-MDT0000 complete [ 6566.381403] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.412934] LustreError: Skipped 1 previous similar message [ 6571.580248] LustreError: 220898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207035 with bad export cookie 8231130246409695647 [ 6571.581235] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6571.606934] LustreError: 220898:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6571.858226] Lustre: server umount lustre-MDT0001 complete [ 6585.152802] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6588.202610] Key type lgssc unregistered [ 6588.468044] LNet: 224628:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6588.490722] LNet: Removed LNI 192.168.204.148@tcp [ 6589.212978] Key type .llcrypt unregistered [ 6589.217057] Key type ._llcrypt unregistered [ 6604.520951] alg: No test for adler32 (adler32-zlib) [ 6605.278995] Key type ._llcrypt registered [ 6605.282528] Key type .llcrypt registered [ 6605.444538] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6606.654089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6607.641975] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 6607.916322] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6607.919641] LNet: Accept secure, port 988 [ 6609.599188] Key type lgssc registered [ 6611.035946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6619.926366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.536843] LustreError: 225905:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 1: rc = -5 [ 6668.255923] LustreError: 225905:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 6698.079403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6702.303599] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6710.404220] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6712.420125] Lustre: server umount lustre-OST0000 complete [ 6727.963797] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6730.521449] Key type lgssc unregistered [ 6730.770566] LNet: 227551:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6730.777920] LNet: Removed LNI 192.168.204.148@tcp [ 6731.558880] Key type .llcrypt unregistered [ 6731.561805] Key type ._llcrypt unregistered [ 6778.136173] alg: No test for adler32 (adler32-zlib) [ 6778.892838] Key type ._llcrypt registered [ 6778.896141] Key type .llcrypt registered [ 6779.036439] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6779.830897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6780.556690] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 6780.782416] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6780.789022] LNet: Accept secure, port 988 [ 6782.471139] Key type lgssc registered [ 6783.805810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6799.200468] loop0: detected capacity change from 0 to 204800000 [ 6799.465465] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6799.505980] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6799.812526] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6799.828305] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6799.920751] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6800.030145] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6800.117123] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6800.272267] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6800.399571] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6800.493227] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.837530] loop0: detected capacity change from 0 to 204800000 [ 6802.064350] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6802.772940] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6803.499440] loop0: detected capacity change from 0 to 204800000 [ 6803.937968] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6804.849734] loop0: detected capacity change from 0 to 204800000 [ 6804.993669] print_req_error: 12 callbacks suppressed [ 6804.993683] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6805.002060] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6805.153648] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6805.239790] blk_update_request: operation not supported error, dev loop0, sector 58272 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6805.418606] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6806.403906] loop0: detected capacity change from 0 to 204800000 [ 6806.447375] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6810.298959] loop0: detected capacity change from 0 to 204800000 [ 6810.333092] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6814.128365] loop0: detected capacity change from 0 to 204800000 [ 6814.159718] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6817.626952] loop0: detected capacity change from 0 to 204800000 [ 6817.683720] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6821.278048] loop0: detected capacity change from 0 to 204800000 [ 6821.345684] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6821.398951] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.455540] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6821.468802] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6821.485062] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6821.498367] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6821.512217] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6821.524194] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6821.537625] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6821.549819] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6821.562066] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6821.569708] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6821.576624] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6821.584815] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6821.594781] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6821.604848] Lustre: 231031:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6833.631461] Lustre: 231310:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776207291/real 1776207291] req@00000000f9af0b3c x1862488287084672/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776207297 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6833.682481] Lustre: server umount MGS complete [ 6834.695481] loop0: detected capacity change from 0 to 204800000 [ 6834.789887] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.286952] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.341902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6835.395766] LustreError: 231364:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 6836.485757] loop1: detected capacity change from 0 to 204800000 [ 6836.645227] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6836.807586] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.876279] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6836.884595] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6836.898027] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6836.915832] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6836.935239] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6836.943685] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6836.957229] Lustre: 231479:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6836.976693] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6837.297807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6837.368898] LustreError: 231479:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 6838.214566] loop2: detected capacity change from 0 to 204800000 [ 6838.381506] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6838.500194] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6838.516181] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 6838.595651] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6838.608783] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6838.627330] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6838.641550] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6838.649723] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6838.660339] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6838.674402] Lustre: 231563:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6838.794949] Lustre: Found index 0 for lustre-OST0000, updating log [ 6838.885445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6838.906019] LustreError: 231563:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 6838.932290] LustreError: 231422:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 6839.814417] loop3: detected capacity change from 0 to 204800000 [ 6840.014809] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6840.092331] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6840.098330] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 6840.159792] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6840.172584] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6840.186117] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6840.202666] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6840.232933] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6840.246697] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6840.255298] Lustre: 231648:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6840.279984] Lustre: Found index 1 for lustre-OST0001, updating log [ 6840.409587] LustreError: 231648:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 6841.428066] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 6841.437243] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 6842.476491] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 6842.481130] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 6847.887386] LustreError: 231663:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 5533us: rc = -2 [ 6848.410981] LustreError: 231390:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 575us: rc = -2 [ 6848.432014] LustreError: 231390:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 6862.818126] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6862.827738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.839667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6863.332596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.349645] Lustre: Skipped 2 previous similar messages [ 6863.354777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6863.359516] Lustre: Skipped 3 previous similar messages [ 6866.650164] Lustre: server umount lustre-MDT0000 complete [ 6867.673166] LustreError: 231371:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207331 with bad export cookie 7537033936404324328 [ 6867.694385] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.447908] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.448652] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6868.449088] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6868.476034] Lustre: Skipped 1 previous similar message [ 6868.504026] LustreError: Skipped 1 previous similar message [ 6872.486664] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6872.488576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6872.511284] LustreError: Skipped 1 previous similar message [ 6872.525746] Lustre: Skipped 2 previous similar messages [ 6874.207057] Lustre: server umount lustre-MDT0001 complete [ 6881.358928] Lustre: server umount lustre-OST0000 complete [ 6888.563702] Lustre: server umount lustre-OST0001 complete [ 6897.085808] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 18:55:58 (1776207358) [ 6924.603281] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6934.176909] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6945.038740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6952.348651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6960.395581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6968.174821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6977.616223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6977.680264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.915858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6977.974220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6978.167565] Lustre: lustre-MDT0000: new disk, initializing [ 6978.232069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6978.241034] Lustre: Skipped 1 previous similar message [ 6978.277938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6984.456229] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 6997.636628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6997.701175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.802906] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6997.861420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6997.866256] Lustre: Skipped 1 previous similar message [ 6997.950747] Lustre: lustre-MDT0001: new disk, initializing [ 6998.109975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6998.159454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6998.165594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7003.459577] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7014.210321] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7022.400483] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7032.563756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7032.660189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.673742] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7033.246786] Lustre: lustre-OST0000: new disk, initializing [ 7033.255100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7033.417754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.987969] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7039.582622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7039.593263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7049.848545] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7056.089524] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 7067.138877] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 7071.138932] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 7132.128388] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7132.131094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7132.144146] LustreError: Skipped 1 previous similar message [ 7132.144811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7132.144815] Lustre: Skipped 2 previous similar messages [ 7132.189150] Lustre: Skipped 1 previous similar message [ 7142.376468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7142.388886] Lustre: Skipped 4 previous similar messages [ 7143.391522] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7143.596521] Lustre: server umount lustre-OST0000 complete [ 7151.584266] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7151.591625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7162.335464] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7162.490242] Lustre: server umount lustre-MDT0000 complete [ 7162.853532] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.872206] LustreError: Skipped 3 previous similar messages [ 7166.372758] LustreError: 239122:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207630 with bad export cookie 7537033936404326680 [ 7166.377055] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7166.389410] LustreError: 239122:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7177.294416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.528370] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7194.081389] Lustre: 241182:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776207651/real 1776207651] req@000000009d2cbef8 x1862488287159232/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776207657 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 7205.955070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.423597] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.463740] LustreError: 241831:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7206.524594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.942691] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7217.138948] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.167251] LustreError: Skipped 1 previous similar message [ 7219.956560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.808902] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7234.658077] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7242.767772] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7249.838731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.200399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.206142] Lustre: Skipped 1 previous similar message [ 7255.565848] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7264.812180] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7302.625853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.627184] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7302.627498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7302.627503] Lustre: Skipped 5 previous similar messages [ 7302.641852] Lustre: Skipped 2 previous similar messages [ 7314.400393] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7314.622425] Lustre: server umount lustre-OST0000 complete [ 7314.628570] Lustre: Skipped 2 previous similar messages [ 7322.604315] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7322.617881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.348636] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7337.723684] LustreError: 241800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776207801 with bad export cookie 7537033936404328591 [ 7337.737062] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.741044] LustreError: 241800:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7352.356136] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7355.508523] Key type lgssc unregistered [ 7355.880857] LNet: 245943:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7355.889811] LNet: Removed LNI 192.168.204.148@tcp [ 7356.716903] Key type .llcrypt unregistered [ 7356.719301] Key type ._llcrypt unregistered [ 7400.371848] alg: No test for adler32 (adler32-zlib) [ 7401.129426] Key type ._llcrypt registered [ 7401.133369] Key type .llcrypt registered [ 7401.284599] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7418.958639] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7420.261325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7420.972296] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 7421.214890] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7421.217286] LNet: Accept secure, port 988 [ 7422.895156] Key type lgssc registered [ 7424.339286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7435.536451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7442.992715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7450.065796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7457.211982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7466.218713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7466.356645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.710602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7467.725490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7467.791449] Lustre: lustre-MDT0000: new disk, initializing [ 7467.906670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7467.930562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7473.791513] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7484.891339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7484.947319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.064882] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7485.200840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7485.209319] Lustre: Skipped 1 previous similar message [ 7485.306331] Lustre: lustre-MDT0001: new disk, initializing [ 7485.392921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7485.448325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7485.466139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7489.742231] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7500.244158] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7506.544541] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7513.891843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7514.000732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7514.008985] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7514.418786] Lustre: lustre-OST0000: new disk, initializing [ 7514.443816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7514.644752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.894421] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7520.803122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7520.814213] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7531.109297] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7537.619951] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 7550.622711] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 7556.294981] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 7603.172795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.176152] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7603.182108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7603.183252] Lustre: Skipped 1 previous similar message [ 7605.773245] Lustre: server umount lustre-OST0000 complete [ 7613.408491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.412289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7613.425157] Lustre: Skipped 1 previous similar message [ 7613.431458] Lustre: Skipped 1 previous similar message [ 7615.546111] Lustre: server umount lustre-MDT0000 complete [ 7618.531537] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7618.547480] LustreError: Skipped 1 previous similar message [ 7619.225245] LustreError: 253394:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208083 with bad export cookie 3573776781839530735 [ 7619.231894] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7619.244203] LustreError: 253394:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7619.549211] Lustre: server umount lustre-MDT0001 complete [ 7628.179489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.293186] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7644.128032] Lustre: 255498:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776208102/real 1776208102] req@000000000840d0d0 x1862488959272256/t0(0) o251->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776208108 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7644.173859] Lustre: server umount MGS complete [ 7657.424934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7657.900181] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.921661] LustreError: 256133:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7658.003306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.778970] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7663.078348] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.196373] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.300130] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7672.554316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.143172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7678.533986] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7690.081167] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7698.101881] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7705.303421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7705.688145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.447148] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7718.696703] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7744.992923] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.997674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.011085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7745.024120] Lustre: Skipped 1 previous similar message [ 7745.035587] Lustre: Skipped 2 previous similar messages [ 7746.399836] Lustre: server umount lustre-OST0000 complete [ 7755.233554] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7755.236904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.252459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.265929] Lustre: Skipped 1 previous similar message [ 7755.277907] Lustre: Skipped 1 previous similar message [ 7756.538725] Lustre: server umount lustre-MDT0000 complete [ 7760.283545] LustreError: 256103:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208224 with bad export cookie 3573776781839532632 [ 7760.303636] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7760.304542] LustreError: 256103:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7771.957325] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7774.540804] Key type lgssc unregistered [ 7774.859598] LNet: 260242:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7774.867669] LNet: Removed LNI 192.168.204.148@tcp [ 7775.631491] Key type .llcrypt unregistered [ 7775.633692] Key type ._llcrypt unregistered [ 7803.606506] alg: No test for adler32 (adler32-zlib) [ 7804.378203] Key type ._llcrypt registered [ 7804.393062] Key type .llcrypt registered [ 7804.562957] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7805.901546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.522732] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 7806.720623] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7806.724690] LNet: Accept secure, port 988 [ 7808.407760] Key type lgssc registered [ 7809.545159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7818.509512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.009288] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7820.107200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.378345] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.674169] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7830.506745] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.619933] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.195476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.673033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7841.315925] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7850.453385] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7856.465619] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7863.167814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.623290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7868.833480] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7877.925623] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7885.054645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7885.137854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7885.165475] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 7885.323279] Lustre: lustre-OST0001: new disk, initializing [ 7885.329832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 7885.401398] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7890.536647] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 7893.558274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7893.575237] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7893.663112] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7894.690626] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7894.708702] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 7895.778902] LustreError: 265158:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7895.789188] LustreError: 265158:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7897.826318] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7897.857194] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 7902.000953] LustreError: 265158:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 7902.011054] LustreError: 265158:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 7902.257450] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 7902.612444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7910.307363] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 7910.323385] LustreError: 265153:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 7910.407977] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e0bc78ae800.ost_server_uuid 40 [ 7912.546961] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e0bc78ae800.ost_server_uuid in FULL state after 0 sec [ 7919.611122] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e0bc78ae800.ost_server_uuid 40 [ 7921.296495] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e0bc78ae800.ost_server_uuid in IDLE state after 0 sec [ 7937.580887] Lustre: lustre-OST0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 7939.041140] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7939.041906] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.053317] LustreError: Skipped 1 previous similar message [ 7939.055552] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7939.063417] Lustre: Skipped 1 previous similar message [ 7939.090981] Lustre: Skipped 1 previous similar message [ 7940.076497] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7940.082430] Lustre: Skipped 1 previous similar message [ 7942.654937] Lustre: server umount lustre-OST0001 complete [ 7942.672015] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.769590] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7948.775111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.793792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.323333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.654672] Lustre: server umount lustre-OST0000 complete [ 7959.520648] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7959.530553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.544760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7959.555942] Lustre: Skipped 2 previous similar messages [ 7964.533672] Lustre: server umount lustre-MDT0000 complete [ 7965.665425] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.680760] LustreError: Skipped 5 previous similar messages [ 7969.050103] LustreError: 264727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208432 with bad export cookie 13521777788762844976 [ 7969.057476] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7969.067336] LustreError: 264727:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7969.586937] Lustre: server umount lustre-MDT0001 complete [ 7983.098831] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7986.348400] Key type lgssc unregistered [ 7986.649463] LNet: 267469:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7986.656969] LNet: Removed LNI 192.168.204.148@tcp [ 7987.505494] Key type .llcrypt unregistered [ 7987.508984] Key type ._llcrypt unregistered [ 8030.202219] alg: No test for adler32 (adler32-zlib) [ 8030.958268] Key type ._llcrypt registered [ 8030.961519] Key type .llcrypt registered [ 8031.096259] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8032.151729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8032.887923] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 8033.123065] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 8033.132941] LNet: Accept secure, port 988 [ 8034.831188] Key type lgssc registered [ 8036.308877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8045.727142] loop0: detected capacity change from 0 to 1099511627776 [ 8046.185714] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 8046.189964] *** VALIDATE xfs *** [ 8046.198312] XFS (loop0): Mounting V5 Filesystem [ 8046.214285] XFS (loop0): Ending clean mount [ 8047.679138] loop1: detected capacity change from 0 to 18691697672192 [ 8141.036470] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8160.356800] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8171.113723] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8182.619794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8189.954236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8196.648191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8204.200209] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8213.148417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8213.262952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8214.407612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8214.429070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 8214.482907] Lustre: lustre-MDT0000: new disk, initializing [ 8214.553287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8214.575000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8219.697553] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8231.164478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8231.311756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.465490] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8231.532848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 8231.542436] Lustre: Skipped 1 previous similar message [ 8231.626556] Lustre: lustre-MDT0001: new disk, initializing [ 8231.727462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8231.776831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8231.793850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8237.521796] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8249.925599] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8257.403624] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8265.782967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8265.854695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8265.861806] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 8266.141438] Lustre: lustre-OST0000: new disk, initializing [ 8266.145766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 8266.246274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.507542] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8272.492306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8272.508397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8282.465946] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8288.140894] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 8288.382155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8293.703654] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 8293.909689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8298.464759] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8298.473781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8298.498244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8298.504398] Lustre: Skipped 1 previous similar message [ 8303.589460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8303.597377] Lustre: Skipped 1 previous similar message [ 8308.715167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8308.737283] Lustre: Skipped 1 previous similar message [ 8310.239657] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 8310.502535] Lustre: server umount lustre-OST0000 complete [ 8318.951217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8318.952580] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8318.971658] Lustre: Skipped 1 previous similar message [ 8318.975102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8318.991299] LustreError: Skipped 1 previous similar message [ 8319.004188] Lustre: Skipped 1 previous similar message [ 8323.510811] Lustre: server umount lustre-MDT0000 complete [ 8324.064780] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8327.470791] LustreError: 272443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208791 with bad export cookie 3454227766038264289 [ 8327.473980] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8327.498100] LustreError: 272443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8327.728209] Lustre: server umount lustre-MDT0001 complete [ 8337.530611] XFS (loop0): Unmounting Filesystem [ 8342.700209] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 19:20:04 (1776208804) [ 8350.141089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8350.626460] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8350.637234] LustreError: Skipped 1 previous similar message [ 8350.717191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8355.819061] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.273318] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8360.934627] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8365.539447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.045592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8370.763537] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8379.208218] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8385.688707] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8393.301402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8393.659501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8398.921314] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8408.086923] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8421.345244] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8421.354987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.369093] Lustre: Skipped 1 previous similar message [ 8421.374800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8424.996936] Lustre: server umount lustre-OST0000 complete [ 8432.609603] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8432.609828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.618076] LustreError: Skipped 1 previous similar message [ 8432.620105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.620111] Lustre: Skipped 2 previous similar messages [ 8432.673808] Lustre: Skipped 2 previous similar messages [ 8434.755690] Lustre: server umount lustre-MDT0000 complete [ 8437.728413] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.745911] LustreError: Skipped 1 previous similar message [ 8438.197260] LustreError: 277133:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776208902 with bad export cookie 3454227766038265136 [ 8438.200066] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8438.209795] LustreError: 277133:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8454.998934] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 8459.477275] Key type lgssc unregistered [ 8459.898788] LNet: 280909:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8459.909092] LNet: Removed LNI 192.168.204.148@tcp [ 8461.131407] Key type .llcrypt unregistered [ 8461.137207] Key type ._llcrypt unregistered [ 8494.711501] alg: No test for adler32 (adler32-zlib) [ 8495.477032] Key type ._llcrypt registered [ 8495.481374] Key type .llcrypt registered [ 8495.703929] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8497.214513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8497.651482] Lustre: Lustre: Build Version: 2.15.8_1_g00e7d23 [ 8497.902099] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 8497.907369] LNet: Accept secure, port 988 [ 8499.583191] Key type lgssc registered [ 8501.109125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.509902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.950734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8511.020109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.063946] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8517.020905] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8521.201131] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8525.291141] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.447188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.805737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8532.967546] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8541.670927] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8548.764406] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8557.168508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8557.630110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.075458] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8571.649589] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8578.593263] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 8579.040368] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8579.048603] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.608466] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.610134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8581.630910] Lustre: Skipped 2 previous similar messages [ 8583.689219] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 8584.707151] Lustre: server umount lustre-MDT0001 complete [ 8586.721794] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8586.748368] LustreError: Skipped 1 previous similar message [ 8588.082555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8596.555720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8596.711609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8596.719694] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8596.757495] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8596.967823] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8596.976026] Lustre: Skipped 1 previous similar message [ 8597.146299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.612315] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8602.635285] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8602.644396] Lustre: Skipped 1 previous similar message [ 8602.860456] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8605.700315] LustreError: 286256:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8605.723487] LustreError: 286256:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8605.740964] LustreError: 286256:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8605.757238] LustreError: 282544:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8605.847682] LustreError: 286263:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -17 [ 8605.867384] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 8605.867384] [ 8909.792925] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8909.810060] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.822175] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8912.866173] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8914.932594] Lustre: server umount lustre-MDT0001 complete [ 8916.481242] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8916.498393] LustreError: Skipped 4 previous similar messages [ 8919.213056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8927.109448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8927.262447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8927.265108] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8927.302688] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8927.618680] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8931.836197] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 8932.844236] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8932.856912] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 8936.452614] LustreError: 294535:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 8936.463630] LustreError: 294535:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 8936.476424] LustreError: 294535:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 8936.492296] LustreError: 294535:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 8936.503579] LustreError: 282544:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 8936.519468] LustreError: 282544:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 8936.574807] LustreError: 294538:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 8936.584276] LustreError: 294538:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -17 [ 8936.593989] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.148@tcp [ 8936.593989] [ 9239.099794] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 9239.113031] Lustre: Skipped 3 previous similar messages [ 9240.040773] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9240.049621] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9251.295410] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9251.452614] Lustre: server umount lustre-MDT0001 complete [ 9254.418120] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.448617] LustreError: Skipped 6 previous similar messages [ 9255.288618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9263.670608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.926890] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9263.948125] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9264.028351] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9264.423065] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 9269.451993] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 9269.743102] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9269.765890] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 9269.778574] Lustre: Skipped 2 previous similar messages [ 9273.861674] LustreError: 303009:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 9273.893152] LustreError: 303009:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9273.907094] LustreError: 303009:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9273.916892] LustreError: 282544:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9273.938930] LustreError: 282544:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9274.126413] LustreError: 303012:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 9274.155240] LustreError: 303012:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -114 [ 9274.168644] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 9274.168644] [ 9592.291397] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.294821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9592.300560] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9592.312489] Lustre: Skipped 2 previous similar messages [ 9592.333027] Lustre: Skipped 10 previous similar messages [ 9597.530901] Lustre: server umount lustre-MDT0000 complete [ 9600.754547] LustreError: 285253:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210064 with bad export cookie 15076302895858178987 [ 9600.759916] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9600.762043] LustreError: 285253:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9602.532964] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9602.555186] LustreError: Skipped 5 previous similar messages [ 9615.327160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 9615.574559] Lustre: server umount lustre-MDT0001 complete [ 9628.677970] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9636.516339] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9646.300473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9652.822300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9660.941766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9669.372303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9677.251786] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:42:18 (1776210138) [ 9684.136261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9684.247457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.525122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9684.568877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 9684.666448] Lustre: lustre-MDT0000: new disk, initializing [ 9684.795720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9684.829231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9690.308325] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 9704.898924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9705.051133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.380939] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9705.528931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 9705.545940] Lustre: Skipped 1 previous similar message [ 9705.755239] Lustre: lustre-MDT0001: new disk, initializing [ 9706.055576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9706.171174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9706.191485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9711.854690] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 9723.205634] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9730.396535] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9738.085066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9738.158676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9738.163776] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 9738.468671] Lustre: lustre-OST0000: new disk, initializing [ 9738.475798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 9738.571706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.601928] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all 8 [ 9745.480080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9745.501196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9753.814730] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9761.248155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9761.255229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9761.265053] Lustre: Skipped 3 previous similar messages [ 9761.279948] Lustre: Skipped 9 previous similar messages [ 9763.023649] Lustre: server umount lustre-MDT0000 complete [ 9763.036087] Lustre: Skipped 1 previous similar message [ 9766.378812] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.402226] LustreError: Skipped 5 previous similar messages [ 9766.798653] LustreError: 314499:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 9766.808210] LustreError: 314499:0:(llog_cat.c:739:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 9766.810940] LustreError: 313629:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776210230 with bad export cookie 15076302895858180898 [ 9766.814833] LustreError: 166-1: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9766.820828] LustreError: 314499:0:(llog_cat.c:775:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 9766.867263] LustreError: 313629:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9777.631535] Lustre: 316888:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776210235/real 1776210235] req@000000006925099c x1862490087681280/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776210241 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 9810.941149] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 9821.968551] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 9837.425288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9848.073265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9855.859787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9864.247401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9866.207923] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9610 sec ========== 19:45:27 (1776210327) [ 9921.367848] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 9924.494219] Key type lgssc unregistered [ 9924.763757] LNet: 322002:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9924.781570] LNet: Removed LNI 192.168.204.148@tcp [ 9925.720368] Key type .llcrypt unregistered [ 9925.723118] Key type ._llcrypt unregistered