[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495520467 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007349] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010119] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.022225] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026853] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030041] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032027] *** VALIDATE tmpfs *** [ 0.033197] *** VALIDATE proc *** [ 0.034248] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.036262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040013] Speculative Store Bypass: Vulnerable [ 0.043291] debug: unmapping init [mem 0xffffffffa8859000-0xffffffffa8860fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054263] rcu: Hierarchical SRCU implementation. [ 0.056427] smp: Bringing up secondary CPUs ... [ 0.057598] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.061066] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139735] node 0 deferred pages initialised in 72ms [ 0.141109] devtmpfs: initialized [ 0.142226] x86/mm: Memory block size: 128MB [ 0.144780] gcov: version magic: 0x41383552 [ 0.145616] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150448] pinctrl core: initialized pinctrl subsystem [ 0.152280] [ 0.152696] ************************************************************* [ 0.155014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157017] ** ** [ 0.158014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161021] ** ** [ 0.163017] ** This means that this kernel is built to expose internal ** [ 0.166022] ** IOMMU data structures, which may compromise security on ** [ 0.169020] ** your system. ** [ 0.171018] ** ** [ 0.173017] ** If you see this message and you are not debugging the ** [ 0.175013] ** kernel, report this immediately to your vendor! ** [ 0.177013] ** ** [ 0.178011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180023] ************************************************************* [ 0.182216] NET: Registered protocol family 16 [ 0.184520] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.189075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193132] cpuidle: using governor menu [ 0.194613] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196649] PCI: Using configuration type 1 for base access [ 0.199153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.208060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214054] cryptd: max_cpu_qlen set to 1000 [ 0.216264] ACPI: Added _OSI(Module Device) [ 0.218017] ACPI: Added _OSI(Processor Device) [ 0.220024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226281] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234054] ACPI: Interpreter enabled [ 0.238104] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240027] ACPI: Using IOAPIC for interrupt routing [ 0.242215] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256454] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265122] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270284] acpiphp: Slot [2] registered [ 0.271188] acpiphp: Slot [3] registered [ 0.273128] acpiphp: Slot [4] registered [ 0.274139] acpiphp: Slot [5] registered [ 0.275178] acpiphp: Slot [6] registered [ 0.277144] acpiphp: Slot [7] registered [ 0.278136] acpiphp: Slot [8] registered [ 0.279128] acpiphp: Slot [9] registered [ 0.282189] acpiphp: Slot [10] registered [ 0.283161] acpiphp: Slot [11] registered [ 0.285143] acpiphp: Slot [12] registered [ 0.286112] acpiphp: Slot [13] registered [ 0.287118] acpiphp: Slot [14] registered [ 0.289111] acpiphp: Slot [15] registered [ 0.290133] acpiphp: Slot [16] registered [ 0.292124] acpiphp: Slot [17] registered [ 0.293164] acpiphp: Slot [18] registered [ 0.295156] acpiphp: Slot [19] registered [ 0.296122] acpiphp: Slot [20] registered [ 0.297118] acpiphp: Slot [21] registered [ 0.299159] acpiphp: Slot [22] registered [ 0.301139] acpiphp: Slot [23] registered [ 0.302128] acpiphp: Slot [24] registered [ 0.304135] acpiphp: Slot [25] registered [ 0.305125] acpiphp: Slot [26] registered [ 0.307133] acpiphp: Slot [27] registered [ 0.308127] acpiphp: Slot [28] registered [ 0.310157] acpiphp: Slot [29] registered [ 0.311140] acpiphp: Slot [30] registered [ 0.313127] acpiphp: Slot [31] registered [ 0.314067] PCI host bridge to bus 0000:00 [ 0.316022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.322033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325031] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.327037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329261] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334726] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338507] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352541] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365603] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368133] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372584] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392023] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.396998] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402927] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432026] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.445960] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475017] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.484847] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.498014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.513025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.521900] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.528017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.535018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.550025] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.560573] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.565014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.571015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.582016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.593430] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.601015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.605013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.616016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.625153] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.629368] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.632359] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.633244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.639033] iommu: Default domain type: Passthrough [ 0.640388] SCSI subsystem initialized [ 0.641132] ACPI: bus type USB registered [ 0.643091] usbcore: registered new interface driver usbfs [ 0.644087] usbcore: registered new interface driver hub [ 0.646090] usbcore: registered new device driver usb [ 0.648174] pps_core: LinuxPPS API ver. 1 registered [ 0.649010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.652056] PTP clock support registered [ 0.653128] EDAC MC: Ver: 3.0.0 [ 0.655129] PCI: Using ACPI for IRQ routing [ 0.657846] NetLabel: Initializing [ 0.658008] NetLabel: domain hash size = 128 [ 0.659010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.661080] NetLabel: unlabeled traffic allowed by default [ 0.663140] vgaarb: loaded [ 0.665263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.667014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.672267] clocksource: Switched to clocksource kvm-clock [ 0.778430] VFS: Disk quotas dquot_6.6.0 [ 0.779965] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782380] *** VALIDATE ramfs *** [ 0.783518] *** VALIDATE hugetlbfs *** [ 0.784837] pnp: PnP ACPI init [ 0.787100] pnp: PnP ACPI: found 6 devices [ 0.803537] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.806757] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.808795] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.810642] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.812918] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.815387] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.817876] NET: Registered protocol family 2 [ 0.820075] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.824622] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.827798] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.834814] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.838323] TCP: Hash tables configured (established 65536 bind 65536) [ 0.841697] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.843962] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.846731] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850041] NET: Registered protocol family 1 [ 0.852772] RPC: Registered named UNIX socket transport module. [ 0.855370] RPC: Registered udp transport module. [ 0.857207] RPC: Registered tcp transport module. [ 0.859054] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861618] NET: Registered protocol family 44 [ 0.863464] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.866030] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.868313] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.870918] PCI: CLS 0 bytes, default 64 [ 0.872639] Unpacking initramfs... [ 2.272445] debug: unmapping init [mem 0xffff8f943cc54000-0xffff8f943ffbffff] [ 2.276518] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.278847] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.281378] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.800933] Initialise system trusted keyrings [ 2.802281] Key type blacklist registered [ 2.804018] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.812744] zbud: loaded [ 2.815535] *** VALIDATE nfs *** [ 2.816481] *** VALIDATE nfs4 *** [ 2.817647] pstore: using deflate compression [ 2.820951] Platform Keyring initialized [ 2.921261] NET: Registered protocol family 38 [ 2.923987] Key type asymmetric registered [ 2.925469] Asymmetric key parser 'x509' registered [ 2.927220] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.930530] io scheduler mq-deadline registered [ 2.932271] io scheduler kyber registered [ 2.933900] io scheduler bfq registered [ 2.936173] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.939077] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.942055] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.944340] ACPI: Power Button [PWRF] [ 3.041052] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.133255] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.319499] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.416119] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.606906] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.640115] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.672597] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.678347] Non-volatile memory driver v1.3 [ 3.681840] Linux agpgart interface v0.103 [ 3.710667] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.713152] vda: detected capacity change from 0 to 34816000 [ 3.729602] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.733453] vdb: detected capacity change from 0 to 1073741824 [ 3.749106] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.752192] vdc: detected capacity change from 0 to 2621440000 [ 3.766962] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.771115] vdd: detected capacity change from 0 to 2621440000 [ 3.784817] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.787270] vde: detected capacity change from 0 to 4294967296 [ 3.805826] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.808746] vdf: detected capacity change from 0 to 4294967296 [ 3.815339] libphy: Fixed MDIO Bus: probed [ 3.824726] usbcore: registered new interface driver usbserial_generic [ 3.828095] usbserial: USB Serial support registered for generic [ 3.831348] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.836661] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.839042] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.841904] mousedev: PS/2 mouse device common for all mice [ 3.845465] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.848341] rtc_cmos 00:05: RTC can wake from S4 [ 3.857321] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.861989] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.865668] rtc_cmos 00:05: registered as rtc0 [ 3.869303] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.872750] intel_pstate: CPU model not supported [ 3.876232] hid: raw HID events driver (C) Jiri Kosina [ 3.878921] usbcore: registered new interface driver usbhid [ 3.881860] usbhid: USB HID core driver [ 3.883910] drop_monitor: Initializing network drop monitor service [ 3.886315] Initializing XFRM netlink socket [ 3.888585] NET: Registered protocol family 10 [ 3.891365] Segment Routing with IPv6 [ 3.893635] NET: Registered protocol family 17 [ 3.896649] mpls_gso: MPLS GSO support [ 3.902975] RAS: Correctable Errors collector initialized. [ 3.905051] AVX version of gcm_enc/dec engaged. [ 3.906465] AES CTR mode by8 optimization enabled [ 3.988300] sched_clock: Marking stable (3988273610, 0)->(4886748585, -898474975) [ 3.992695] registered taskstats version 1 [ 3.994531] Loading compiled-in X.509 certificates [ 3.997571] zswap: loaded using pool lzo/zbud [ 4.032608] Key type big_key registered [ 4.049288] Key type encrypted registered [ 4.051613] ima: No TPM chip found, activating TPM-bypass! [ 4.054207] ima: Allocated hash algorithm: sha1 [ 4.056752] ima: No architecture policies found [ 4.059776] evm: Initialising EVM extended attributes: [ 4.063206] evm: security.selinux [ 4.064980] evm: security.ima [ 4.066636] evm: security.capability [ 4.068510] evm: HMAC attrs: 0x1 [ 4.072296] rtc_cmos 00:05: setting system clock to 2025-12-05 12:16:54 UTC (1764937014) [ 4.078454] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 4.081472] debug: unmapping init [mem 0xffffffffa8582000-0xffffffffa8858fff] [ 4.090089] Write protecting the kernel read-only data: 28672k [ 4.094032] debug: unmapping init [mem 0xffffffffa6c03000-0xffffffffa6dfffff] [ 4.096817] debug: unmapping init [mem 0xffffffffa7514000-0xffffffffa75fffff] [ 4.132462] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.139991] systemd[1]: Detected virtualization kvm. [ 4.142687] systemd[1]: Detected architecture x86-64. [ 4.145685] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.174271] systemd[1]: No hostname configured. [ 4.176925] systemd[1]: Set hostname to . [ 4.179880] random: systemd: uninitialized urandom read (16 bytes read) [ 4.183255] systemd[1]: Initializing machine ID from random generator. [ 4.229266] random: ln: uninitialized urandom read (6 bytes read) [ 4.311175] random: systemd: uninitialized urandom read (16 bytes read) [ 4.314113] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.319704] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.324915] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.918816] device-mapper: uevent: version 1.0.3 [ 4.920605] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.614576] virtio_net virtio0 ens2: renamed from eth0 [ 5.619388] random: fast init done [ 5.678711] scsi host0: ata_piix [ 5.698550] scsi host1: ata_piix [ 5.700301] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.702965] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.230972] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.573995] random: crng init done [ 10.575395] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.847547] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.921142] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.166941] SELinux: Disabled at runtime. [ 12.231904] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.240434] systemd[1]: Detected virtualization kvm. [ 12.242292] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.721689] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.724927] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.729296] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.735267] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.739802] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.757876] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.762335] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 12.883832] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.141855] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.398140] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.403278] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.515109] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.535218] EDAC sbridge: Ver: 1.1.2 [ 15.164509] Key type dns_resolver registered [ 15.455845] NFS: Registering the id_resolver key type [ 15.458074] Key type id_resolver registered [ 15.460351] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg342-server login: [ 38.230313] libcfs: loading out-of-tree module taints kernel. [ 38.267080] alg: No test for adler32 (adler32-zlib) [ 39.022431] Key type ._llcrypt registered [ 39.031767] Key type .llcrypt registered [ 39.146730] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 55.288641] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 56.722588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.192550] hrtimer: interrupt took 6542103 ns [ 57.368477] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 58.313510] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 58.326681] LNet: Accept secure, port 988 [ 60.111379] Key type lgssc registered [ 61.556322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.281787] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 83.183244] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.019183] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.446543] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.571418] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.988894] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 129.973306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.062859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.256624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 131.297534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 131.392238] Lustre: lustre-MDT0000: new disk, initializing [ 131.466573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.491294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 135.005051] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 147.136954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.198438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.254195] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 147.274806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 147.278623] Lustre: Skipped 1 previous similar message [ 147.351415] Lustre: lustre-MDT0001: new disk, initializing [ 147.406182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 147.434195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 147.438954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 150.606502] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 161.865136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.937405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 161.942234] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 162.090554] Lustre: lustre-OST0000: new disk, initializing [ 162.102894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 162.160500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.312500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 164.317242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 165.465406] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 176.646545] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.695495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.699903] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 176.799762] Lustre: lustre-OST0001: new disk, initializing [ 176.802823] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 176.868320] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.150428] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 182.274433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 182.278454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 188.864452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.652685] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 204.120981] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing check_logdir /tmp/testlogs/ [ 207.869894] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing yml_node [ 211.145380] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 213.219646] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 214.830659] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 215.937518] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Dec 5 07:20:25 EST 2025 [ 221.068564] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 222.282217] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 238.560532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 238.561493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 238.566392] Lustre: Skipped 1 previous similar message [ 242.492788] Lustre: server umount lustre-MDT0000 complete [ 243.681995] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 243.688594] LustreError: Skipped 3 previous similar messages [ 245.132438] LustreError: 9218:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937255 with bad export cookie 14493237798964839933 [ 245.132975] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.159700] LustreError: 9218:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 248.799670] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 248.799875] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 248.802380] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 248.802389] Lustre: Skipped 3 previous similar messages [ 248.807328] LustreError: Skipped 1 previous similar message [ 248.811970] Lustre: Skipped 3 previous similar messages [ 250.849915] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.850643] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 250.883710] Lustre: Skipped 2 previous similar messages [ 251.582729] Lustre: server umount lustre-MDT0001 complete [ 254.860363] Lustre: server umount lustre-OST0000 complete [ 258.268172] Lustre: server umount lustre-OST0001 complete [ 263.505533] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 270.621627] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 278.505610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 284.520656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 289.970622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 295.341313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.712603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.772161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 300.935884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 300.956675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 301.003986] Lustre: lustre-MDT0000: new disk, initializing [ 301.086936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 301.103895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 304.305921] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 312.154610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.208975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.269284] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 312.298935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 312.302143] Lustre: Skipped 1 previous similar message [ 312.349602] Lustre: lustre-MDT0001: new disk, initializing [ 312.408562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 312.444093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 312.448464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 315.585441] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 323.792024] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 328.608480] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 334.592326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.686997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.690658] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 334.925983] Lustre: lustre-OST0000: new disk, initializing [ 334.928378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 334.992035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.509395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 336.523364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 339.118567] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 347.967193] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 351.525157] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 351.716707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 355.092517] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 355.281588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 357.346528] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 357.350571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.364647] LustreError: Skipped 1 previous similar message [ 357.365194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 357.374634] Lustre: Skipped 1 previous similar message [ 362.473833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 362.484088] Lustre: Skipped 2 previous similar messages [ 371.167321] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 371.400522] Lustre: server umount lustre-OST0000 complete [ 377.826208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.852789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 377.855860] Lustre: Skipped 2 previous similar messages [ 380.915108] Lustre: server umount lustre-MDT0000 complete [ 382.944391] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.950133] LustreError: Skipped 2 previous similar messages [ 384.030423] LustreError: 14326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937394 with bad export cookie 14493237798964841396 [ 384.033602] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 384.040709] LustreError: 14326:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 388.900586] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 07:23:18 (1764937398) [ 396.073953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.441144] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 396.509967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.582778] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 407.010768] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.041147] LustreError: Skipped 1 previous similar message [ 409.290371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.819662] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 421.040550] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 427.481370] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 433.499972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 437.726546] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 445.503886] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 454.369770] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 460.768457] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 460.777314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.789929] Lustre: Skipped 1 previous similar message [ 460.796954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.095274] Lustre: server umount lustre-MDT0000 complete [ 466.097254] Lustre: Skipped 1 previous similar message [ 467.942587] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.951529] LustreError: Skipped 2 previous similar messages [ 469.576990] LustreError: 18720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937480 with bad export cookie 14493237798964842250 [ 469.580659] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.590324] LustreError: 18720:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 478.687884] Lustre: 21845:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764937483/real 1764937483] req@00000000565827dd x1850670651094464/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764937489 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 496.386545] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 505.718369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.177268] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.231248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.234587] Lustre: Skipped 2 previous similar messages [ 510.643142] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 518.711633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.918996] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 531.114361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.564756] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 544.144690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 544.268450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 544.279536] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 544.461326] Lustre: lustre-OST0001: new disk, initializing [ 544.466677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 546.457545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 546.469688] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 548.752249] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 558.150986] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 561.755153] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 575.459593] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.464074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.475887] Lustre: Skipped 2 previous similar messages [ 575.479130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 575.481791] Lustre: Skipped 3 previous similar messages [ 589.279106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 589.419726] Lustre: server umount lustre-MDT0000 complete [ 589.428938] Lustre: Skipped 2 previous similar messages [ 592.867040] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.880647] LustreError: Skipped 5 previous similar messages [ 593.030720] LustreError: 23209:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937603 with bad export cookie 14493237798964843097 [ 593.039897] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.085705] LustreError: 23209:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 603.104974] Lustre: 27040:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764937607/real 1764937607] req@00000000191f168f x1850670651125824/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764937613 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 605.089460] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764937608/real 1764937608] req@00000000d441b406 x1850670651125952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764937615 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 605.110315] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.119205] Lustre: Skipped 3 previous similar messages [ 620.549488] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 631.012810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.778611] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 643.526841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.977740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 643.986350] Lustre: Skipped 4 previous similar messages [ 647.863971] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 656.064279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.536540] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 668.821101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.967725] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 680.811968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 684.243104] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 684.246949] Lustre: Skipped 1 previous similar message [ 694.755944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.761617] Lustre: Skipped 2 previous similar messages [ 694.763959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.779127] Lustre: Skipped 14 previous similar messages [ 699.871165] Lustre: server umount lustre-MDT0000 complete [ 699.883542] Lustre: Skipped 3 previous similar messages [ 703.564297] LustreError: 28453:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937713 with bad export cookie 14493237798964844343 [ 703.576995] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.591061] LustreError: 28453:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 712.095331] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764937715/real 1764937715] req@00000000eddf136f x1850670651150656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764937722 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 712.140390] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 716.259924] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764937719/real 1764937719] req@00000000d9a426b9 x1850670651150848/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764937726 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 716.293364] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 726.781747] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 07:28:55 (1764937735) [ 752.962272] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 762.568789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.949743] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 762.965964] LustreError: Skipped 12 previous similar messages [ 766.511729] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 774.591723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.549246] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 787.267432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 791.810123] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 800.965923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.617872] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 811.645719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 815.242466] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 826.340323] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 826.348585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.372156] Lustre: Skipped 2 previous similar messages [ 826.385894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.951140] Lustre: server umount lustre-MDT0000 complete [ 830.953889] Lustre: Skipped 3 previous similar messages [ 834.834386] LustreError: 37311:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937845 with bad export cookie 14493237798964845414 [ 834.840151] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.127315] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764937847/real 1764937847] req@00000000bb9365fd x1850670651175744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764937854 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 844.187360] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 858.586962] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 07:31:07 (1764937867) [ 871.836601] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 881.397188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.623342] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 894.297721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.809831] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 907.155468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.664444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.673919] Lustre: Skipped 8 previous similar messages [ 911.968245] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 920.477148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 925.402664] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 933.219423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 946.144100] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 955.559387] LustreError: 39487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764937965 with bad export cookie 14493237798964846485 [ 955.565660] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 955.575773] LustreError: 39487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 956.898728] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.917528] Lustre: Skipped 8 previous similar messages [ 970.243571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.290440] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xc92252fd5c905f95 to 0xc92252fd5c9063fc [ 973.316158] Lustre: MGC192.168.203.142@tcp: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 973.668820] BUG: spinlock bad magic on CPU#2, lctl/43422 [ 973.672871] lock: 0xffff8f94bf966900, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 973.685328] CPU: 2 PID: 43422 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 973.689222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 973.695660] Call Trace: [ 973.697946] ? dump_stack+0xbb/0x10e [ 973.703245] ? spin_dump.cold.3+0x32/0x65 [ 973.704871] ? do_raw_spin_lock+0xce/0x190 [ 973.708399] ? _raw_spin_lock+0x21/0x30 [ 973.710432] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 973.722174] ? __seccomp_filter+0x4b/0x680 [ 973.723747] ? full_proxy_write+0x5e/0xa0 [ 973.725213] ? __vfs_write+0x1c/0x60 [ 973.728873] ? vfs_write+0xd8/0x2b0 [ 973.730579] ? ksys_write+0x66/0x120 [ 973.732645] ? __x64_sys_write+0x1e/0x30 [ 973.736001] ? do_syscall_64+0xc1/0x440 [ 973.737271] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 977.788690] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 986.149345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.695674] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 990.643178] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 997.794578] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 999.483515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1004.868475] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1006.570922] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1012.192890] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1020.246011] LustreError: 39487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764938030 with bad export cookie 14493237798964847612 [ 1020.255059] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.260345] LustreError: 39487:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1030.559128] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764938033/real 1764938033] req@00000000df8e3df1 x1850670651220608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764938040 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1030.596104] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1044.469758] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 07:34:13 (1764938053) [ 1057.975919] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1068.528440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.987906] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1069.005119] LustreError: Skipped 39 previous similar messages [ 1073.277415] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1081.175651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.334773] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1092.652549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1096.776296] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1104.682685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1109.048673] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1115.488885] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1118.062722] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1118.066059] Lustre: Skipped 2 previous similar messages [ 1136.119412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.122234] Lustre: Skipped 25 previous similar messages [ 1139.634161] Lustre: server umount lustre-MDT0000 complete [ 1139.636241] Lustre: Skipped 9 previous similar messages [ 1143.378279] LustreError: 47250:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764938153 with bad export cookie 14493237798964848417 [ 1143.383741] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.995581] Lustre: 50840:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764938157/real 1764938157] req@000000009b7b440b x1850670651247744/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764938163 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1153.003730] Lustre: 50840:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1164.306594] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 07:36:13 (1764938173) [ 1166.839582] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1168.505521] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 07:36:17 (1764938177) [ 1178.337935] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1185.195761] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1193.384801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1198.750488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.658995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1204.736507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.012152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1205.096583] Lustre: lustre-MDT0000: new disk, initializing [ 1205.180438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1208.945445] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1217.333349] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1223.134924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1223.214357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.230216] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1227.454266] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1236.684386] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1238.680982] Lustre: DEBUG MARKER: create test files [ 1266.146167] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1266.157889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.166559] Lustre: Skipped 11 previous similar messages [ 1277.407174] Lustre: 55553:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764938281/real 1764938281] req@000000002992abbe x1850670651256512/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764938287 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1277.429436] Lustre: 55553:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1290.209929] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1293.707635] Key type lgssc unregistered [ 1293.996278] LNet: 56156:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.015638] LNet: Removed LNI 192.168.203.142@tcp [ 1295.011444] Key type .llcrypt unregistered [ 1295.013057] Key type ._llcrypt unregistered [ 1322.481780] alg: No test for adler32 (adler32-zlib) [ 1323.236404] Key type ._llcrypt registered [ 1323.237756] Key type .llcrypt registered [ 1323.389270] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1324.693222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.136683] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 1325.352222] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1325.355612] LNet: Accept secure, port 988 [ 1327.023114] Key type lgssc registered [ 1328.219032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.806456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.353636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.270610] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1350.232809] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1357.557643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1357.983117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.469463] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1368.243209] Lustre: lustre-OST0000: deleting orphan objects from 0x0:101 to 0x0:129 [ 1368.372205] Lustre: 59001:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1368.387529] Lustre: 59001:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1368.403616] Lustre: 59001:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1368.407916] Lustre: 59001:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1368.423456] Lustre: 59001:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1368.440493] Lustre: 59001:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1368.815467] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1374.279680] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1379.155632] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1386.607782] Lustre: server umount lustre-OST0000 complete [ 1396.191146] Lustre: 60018:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764938400/real 1764938400] req@0000000003b78e31 x1850671979566784/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764938406 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1401.824292] Lustre: 60018:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764938406/real 1764938406] req@00000000c7d651cc x1850671979567296/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764938412 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1402.012726] Lustre: server umount lustre-MDT0000 complete [ 1411.757928] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1414.286992] Key type lgssc unregistered [ 1414.563845] LNet: 60624:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.574535] LNet: Removed LNI 192.168.203.142@tcp [ 1415.427145] Key type .llcrypt unregistered [ 1415.429802] Key type ._llcrypt unregistered [ 1442.193318] alg: No test for adler32 (adler32-zlib) [ 1442.960355] Key type ._llcrypt registered [ 1442.963978] Key type .llcrypt registered [ 1443.058931] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1443.747208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.310907] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 1444.522424] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1444.526427] LNet: Accept secure, port 988 [ 1446.160149] Key type lgssc registered [ 1447.039122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1453.647491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.114511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.675274] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1465.692500] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1472.417534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.730218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.034988] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1485.045495] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1485.139668] Lustre: 63562:0:(osd_internal.h:1344:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1485.146392] Lustre: 63562:0:(osd_handler.c:1944:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1485.151601] Lustre: 63562:0:(osd_handler.c:1951:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1485.155182] Lustre: 63562:0:(osd_handler.c:1961:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1485.160529] Lustre: 63562:0:(osd_handler.c:1968:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1485.165470] Lustre: 63562:0:(osd_handler.c:1975:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1486.094842] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1491.486369] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1500.655792] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.662790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.695651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.321842] Lustre: server umount lustre-OST0000 complete [ 1514.467510] Lustre: 64526:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764938518/real 1764938518] req@000000008a197f96 x1850672104347904/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764938524 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1514.645147] Lustre: server umount lustre-MDT0000 complete [ 1527.490236] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 1530.413862] Key type lgssc unregistered [ 1530.771948] LNet: 65131:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.787384] LNet: Removed LNI 192.168.203.142@tcp [ 1531.589298] Key type .llcrypt unregistered [ 1531.591140] Key type ._llcrypt unregistered [ 1548.441818] alg: No test for adler32 (adler32-zlib) [ 1549.198939] Key type ._llcrypt registered [ 1549.202728] Key type .llcrypt registered [ 1549.337965] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1561.632123] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1562.798729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.240751] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 1563.517420] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 1563.527914] LNet: Accept secure, port 988 [ 1565.185650] Key type lgssc registered [ 1566.175128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1574.413293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1580.456112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.117530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.920076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1599.303433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1599.465250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.907951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1600.978451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1601.084657] Lustre: lustre-MDT0000: new disk, initializing [ 1601.239521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.272869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1607.058820] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1617.364289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1617.451647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.552598] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1617.592784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1617.595387] Lustre: Skipped 1 previous similar message [ 1617.655077] Lustre: lustre-MDT0001: new disk, initializing [ 1617.764684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1617.834644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1617.839753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1621.428728] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1630.780933] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1635.353972] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1641.676189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1641.736612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.744869] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1641.969877] Lustre: lustre-OST0000: new disk, initializing [ 1641.983968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1642.062714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.633275] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1650.746177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1650.754672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1654.622661] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1658.887859] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1659.086831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1662.875711] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1663.095151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1666.528498] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.530396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.548285] LustreError: Skipped 1 previous similar message [ 1666.551101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.561281] Lustre: Skipped 1 previous similar message [ 1670.932799] Lustre: server umount lustre-OST0000 complete [ 1676.771764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.773428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1676.793026] Lustre: Skipped 2 previous similar messages [ 1676.811621] Lustre: Skipped 1 previous similar message [ 1680.598117] Lustre: server umount lustre-MDT0000 complete [ 1681.887984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.917140] LustreError: Skipped 1 previous similar message [ 1682.934442] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.945590] LustreError: Skipped 1 previous similar message [ 1684.225764] LustreError: 68405:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764938694 with bad export cookie 18189368626920240305 [ 1684.230140] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1684.236898] LustreError: 68405:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1684.604603] Lustre: server umount lustre-MDT0001 complete [ 1693.103641] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 07:45:02 (1764938702) [ 1701.856802] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1713.098069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1735.478256] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 1743.764502] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 1753.343878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1759.819406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1765.720656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1771.993899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1779.369961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1779.442101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.769451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1779.828557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1779.884346] Lustre: lustre-MDT0000: new disk, initializing [ 1779.951572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.976948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1784.163840] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1794.105392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.185466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.248792] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1794.279697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1794.287907] Lustre: Skipped 1 previous similar message [ 1794.414810] Lustre: lustre-MDT0001: new disk, initializing [ 1794.525947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1794.578531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1794.582582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1798.927254] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1809.395100] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1814.951863] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1820.790576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1820.876679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.888806] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1821.174953] Lustre: lustre-OST0000: new disk, initializing [ 1821.184063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1821.290379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.636389] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1826.863632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1826.867158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1834.356469] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1838.067646] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1838.333551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1842.319850] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1842.610573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1847.269652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.270505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1847.284768] Lustre: Skipped 1 previous similar message [ 1847.302407] Lustre: Skipped 1 previous similar message [ 1850.606564] Lustre: server umount lustre-OST0000 complete [ 1855.970773] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1855.976876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.001979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1860.334725] Lustre: server umount lustre-MDT0000 complete [ 1862.629252] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1862.646085] LustreError: Skipped 1 previous similar message [ 1864.088981] LustreError: 76702:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764938874 with bad export cookie 18189368626920241159 [ 1864.096757] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.108540] LustreError: 76702:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1871.931771] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 07:48:01 (1764938881) [ 1877.156926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.496670] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.579137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1880.972206] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1882.604401] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1888.254793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1892.264726] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1899.759562] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1904.880479] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1910.421319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1910.653642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.659147] Lustre: Skipped 1 previous similar message [ 1914.136776] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1920.820966] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1924.086994] Lustre: 83998:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting afb2b97a-32da-44fc-a052-7c0cae67cf76 at adminstrative request [ 1928.101127] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1928.129297] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1934.306673] Lustre: MGS: Client afb2b97a-32da-44fc-a052-7c0cae67cf76 (at 0@lo) reconnecting [ 1934.324489] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xfc6d9e6510373b56 to 0xfc6d9e6510373dfd [ 1934.335540] Lustre: MGC192.168.203.142@tcp: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 1960.669786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1960.763175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.768734] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1960.910642] Lustre: lustre-OST0001: new disk, initializing [ 1960.913947] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1964.876397] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 1969.252918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1969.260635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1973.159456] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1979.885953] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1979.886861] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.916319] Lustre: Skipped 1 previous similar message [ 1979.926299] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1979.930904] Lustre: Skipped 3 previous similar messages [ 1982.273457] Lustre: server umount lustre-OST0001 complete [ 1982.278633] Lustre: Skipped 1 previous similar message [ 1985.010762] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.016461] LustreError: Skipped 1 previous similar message [ 1989.600153] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1989.601989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.610107] LustreError: Skipped 2 previous similar messages [ 1989.619189] Lustre: Skipped 1 previous similar message [ 1989.624557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1989.637152] Lustre: Skipped 1 previous similar message [ 2002.734094] Lustre: server umount lustre-MDT0000 complete [ 2002.738392] Lustre: Skipped 1 previous similar message [ 2005.473950] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.491241] LustreError: Skipped 3 previous similar messages [ 2006.182218] LustreError: 81239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764939016 with bad export cookie 18189368626920242685 [ 2006.194573] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.977200] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 07:50:24 (1764939024) [ 2016.522129] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2018.228959] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 07:50:27 (1764939027) [ 2019.767179] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2021.718426] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 07:50:30 (1764939030) [ 2023.471373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2025.380776] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 07:50:34 (1764939034) [ 2027.155243] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2031.674733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2032.654580] loop: module loaded [ 2032.677671] loop0: detected capacity change from 0 to 4294967296 [ 2032.735873] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2034.181826] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2037.812460] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2041.471820] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2049.235936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2052.300376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2055.845581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2059.006735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2064.947423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2065.081272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.100816] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2065.124077] LustreError: 88446:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2065.136224] LustreError: 88446:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2065.138880] LustreError: 88446:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2065.158452] LustreError: 88446:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2081.703857] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2089.547547] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2097.936137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2103.731338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2109.131589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2115.099524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2121.054650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2121.125975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.299149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2121.330866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2121.422171] Lustre: lustre-MDT0000: new disk, initializing [ 2121.534347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.538024] Lustre: Skipped 1 previous similar message [ 2121.555153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2125.380758] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2134.757946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2134.819075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.905611] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2135.130538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2138.755936] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2147.334623] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2153.660644] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2159.688572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2159.778712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2159.782469] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2163.652148] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2170.432512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2172.305901] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2176.414816] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2176.613621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2180.388845] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2180.606872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2185.703556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.709280] Lustre: Skipped 3 previous similar messages [ 2185.713964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.717022] Lustre: Skipped 4 previous similar messages [ 2188.465507] Lustre: server umount lustre-OST0000 complete [ 2188.467653] Lustre: Skipped 1 previous similar message [ 2206.177211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2207.203263] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.463574] LustreError: 93386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764939219 with bad export cookie 18189368626920243266 [ 2209.464721] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.478755] LustreError: 93386:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2217.485527] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 07:53:46 (1764939226) [ 2223.663268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.064802] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2235.179865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.959390] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2245.607231] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2250.530194] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2256.244514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2256.515419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.518319] Lustre: Skipped 4 previous similar messages [ 2260.161335] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2266.018284] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2271.246541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.307273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2271.311943] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2271.403171] Lustre: lustre-OST0001: new disk, initializing [ 2271.405080] Lustre: Skipped 2 previous similar messages [ 2271.412337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2271.418111] Lustre: Skipped 3 previous similar messages [ 2275.335454] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2280.517696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2280.523448] Lustre: Skipped 2 previous similar messages [ 2280.540437] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2284.347860] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2300.893721] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2319.714549] Lustre: Failing over lustre-MDT0000 [ 2320.069059] Lustre: server umount lustre-MDT0000 complete [ 2320.072780] Lustre: Skipped 2 previous similar messages [ 2321.889072] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.892658] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.894093] Lustre: Skipped 5 previous similar messages [ 2321.898320] LustreError: Skipped 4 previous similar messages [ 2329.055139] Lustre: 66634:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764939332/real 1764939332] req@0000000003499f6d x1850672229241024/t0(0) o400->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764939339 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2329.079385] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.022497] LDISKFS-fs (dm-0): recovery complete [ 2343.028660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.490119] Lustre: Evicted from MGS (at 192.168.203.142@tcp) after server handle changed from 0xfc6d9e6510374398 to 0xfc6d9e651037d182 [ 2346.514554] Lustre: MGC192.168.203.142@tcp: Connection restored to (at 0@lo) [ 2346.786437] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2351.196435] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2351.203654] Lustre: 101012:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2351.488332] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2352.113298] Lustre: 95855:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2352.116510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2352.147242] Lustre: 95855:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2352.236154] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2352.578728] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2352.956666] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2352.970331] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2353.303384] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2353.311485] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2353.330623] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2353.345063] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2354.010710] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2354.024292] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2354.359193] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2354.362284] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages [ 2355.335958] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2355.352427] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 5 previous similar messages [ 2356.339854] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2356.343510] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2356.679154] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2356.684822] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2359.545157] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2359.562755] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2360.559843] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2360.577339] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2360.903157] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2360.906517] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2367.592119] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2367.606970] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 24 previous similar messages [ 2368.592755] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2368.598133] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2368.921771] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2368.927235] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2383.601297] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2383.616654] Lustre: 101035:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 49 previous similar messages [ 2384.888492] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2384.901113] LustreError: 101035:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2385.231113] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2385.234441] LustreError: 101035:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2411.258615] Lustre: 101035:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2411.270402] Lustre: 101035:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2411.301938] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2411.313791] LustreError: 101035:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2411.332416] Lustre: 101035:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@000000005e8135c9 x1850672217617600/t0(8589934781) o101->3aacd7e9-e00a-4283-bfd2-4aad79c36c22@192.168.203.42@tcp:87/0 lens 592/0 e 0 to 0 dl 1764939427 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2411.393980] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2411.395881] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2411.463084] Lustre: Skipped 2 previous similar messages [ 2411.495758] Lustre: lustre-OST0001: deleting orphan objects from 0x0:48 to 0x0:545 [ 2411.501158] Lustre: lustre-OST0000: deleting orphan objects from 0x0:80 to 0x0:577 [ 2422.752763] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2422.755270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.776739] Lustre: Skipped 1 previous similar message [ 2422.780706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.784149] Lustre: Skipped 7 previous similar messages [ 2432.479915] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.721459] Lustre: server umount lustre-MDT0000 complete [ 2449.730489] Lustre: Skipped 2 previous similar messages [ 2450.403890] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.442556] LustreError: Skipped 52 previous similar messages [ 2453.784871] LustreError: 98766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764939464 with bad export cookie 18189368626920280450 [ 2453.797128] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2453.804239] LustreError: 98766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2462.932552] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 07:57:52 (1764939472) [ 2471.160817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.174227] Lustre: *** cfs_fail_loc=197, val=0*** [ 2488.799665] LustreError: 103062:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 1: rc = -5 [ 2499.041263] LustreError: 103062:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 2505.213507] Lustre: *** cfs_fail_loc=197, val=0*** [ 2505.215139] Lustre: Skipped 65 previous similar messages [ 2528.863261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.872988] Lustre: Skipped 2 previous similar messages [ 2533.017821] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2540.216289] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2551.336662] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 07:59:20 (1764939560) [ 2573.867680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.508102] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2587.185329] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2590.464835] LustreError: 104721:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2590.474405] LustreError: 104721:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2590.483605] LustreError: 104721:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 2596.831157] Lustre: 105412:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764939601/real 1764939601] req@000000008657d037 x1850672229274816/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764939607 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2611.328928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2626.388586] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2635.513632] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2645.064742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2650.967462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.408513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2662.263055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.897438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.953535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.191299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2668.220881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2668.316322] Lustre: lustre-MDT0000: new disk, initializing [ 2668.422281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2671.400512] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2680.442621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.495747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.752190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2684.426763] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2692.610462] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2696.896756] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2701.932898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.971343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2701.975372] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2705.818471] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2712.863407] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2715.786598] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2715.950941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2718.960937] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2719.140618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2722.784697] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2722.788031] LustreError: Skipped 1 previous similar message [ 2722.790211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.794802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2722.799945] Lustre: Skipped 6 previous similar messages [ 2722.804505] Lustre: Skipped 9 previous similar messages [ 2726.781217] Lustre: server umount lustre-OST0000 complete [ 2726.787271] Lustre: Skipped 3 previous similar messages [ 2738.144576] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.162240] LustreError: Skipped 5 previous similar messages [ 2739.483455] LustreError: 109051:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764939749 with bad export cookie 18189368626920291118 [ 2739.487376] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2747.108939] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 08:02:36 (1764939756) [ 2755.697118] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 2757.909996] Key type lgssc unregistered [ 2758.140431] LNet: 113743:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.148171] LNet: Removed LNI 192.168.203.142@tcp [ 2758.741197] Key type .llcrypt unregistered [ 2758.742479] Key type ._llcrypt unregistered [ 2772.024745] alg: No test for adler32 (adler32-zlib) [ 2772.777574] Key type ._llcrypt registered [ 2772.780557] Key type .llcrypt registered [ 2772.872263] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2784.324083] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2785.126509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2785.426813] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 2785.580744] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 2785.585316] LNet: Accept secure, port 988 [ 2787.215172] Key type lgssc registered [ 2788.104777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2795.291396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2800.266260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.124454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2809.808300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2814.203286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2819.003173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2819.066144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.220798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2820.250731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2820.331378] Lustre: lustre-MDT0000: new disk, initializing [ 2820.388611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.406799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2823.076791] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2831.438309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2831.514767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.517713] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2831.745963] Lustre: lustre-OST0000: new disk, initializing [ 2831.755256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2831.757328] Lustre: Skipped 1 previous similar message [ 2831.818305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.369765] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2843.891659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2843.985572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.989767] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2844.116799] Lustre: lustre-OST0001: new disk, initializing [ 2844.122278] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2844.182212] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2847.682045] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2857.952478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.966057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.973636] Lustre: Skipped 1 previous similar message [ 2861.412127] Lustre: server umount lustre-MDT0000 complete [ 2865.936670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2869.119067] LustreError: 117228:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764939879 with bad export cookie 728172489606563480 [ 2869.126554] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2869.142064] LustreError: 117228:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2869.204174] Lustre: server umount lustre-OST0000 complete [ 2871.909225] Lustre: server umount lustre-OST0001 complete [ 2878.366893] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 08:04:47 (1764939887) [ 2884.246303] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2888.743719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2889.428513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2893.183057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2893.896364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2899.136135] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 08:05:08 (1764939908) [ 2906.464320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2907.700339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2909.096861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2910.277414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2910.951815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2912.170997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2923.145545] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 2929.398546] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 2935.543670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2940.116366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2944.305216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.959859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2952.050663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2952.086026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2952.264493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2952.291607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2952.346572] Lustre: lustre-MDT0000: new disk, initializing [ 2952.416580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.433210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2955.047584] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2961.674299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2961.726579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.781839] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2961.813449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2961.818532] Lustre: Skipped 1 previous similar message [ 2961.880459] Lustre: lustre-MDT0001: new disk, initializing [ 2961.958494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2962.004610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2962.012125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2964.919164] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2972.351459] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2976.767628] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2981.344198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2981.393909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.406761] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2981.548607] Lustre: lustre-OST0000: new disk, initializing [ 2981.552426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2981.619965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.194350] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 2989.594736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2989.600474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2990.628549] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2993.603959] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2993.774389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2996.023888] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2996.168780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2999.778318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.781387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.781938] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2999.786418] Lustre: Skipped 2 previous similar messages [ 3003.448225] Lustre: server umount lustre-OST0000 complete [ 3007.968409] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.974208] LustreError: Skipped 1 previous similar message [ 3007.978349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.989044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.991875] Lustre: Skipped 1 previous similar message [ 3010.016705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.020376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.033590] Lustre: Skipped 1 previous similar message [ 3011.739309] Lustre: server umount lustre-MDT0000 complete [ 3013.912359] LustreError: 125761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940024 with bad export cookie 728172489606564691 [ 3013.914769] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3013.919037] LustreError: 125761:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3019.076415] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 08:07:08 (1764940028) [ 3022.840500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.109346] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3023.163550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.152539] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3028.451376] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.402925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.773363] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3040.813727] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3044.658666] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3048.914231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.738556] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3056.640778] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3068.935648] Lustre: *** cfs_fail_loc=159, val=0*** [ 3068.937960] Lustre: Skipped 2 previous similar messages [ 3076.223877] Lustre: lustre-MDT0000: Client fa2e8522-28e9-4ae4-aabb-a612e65c9a93 (at 192.168.203.42@tcp) reconnecting [ 3077.012024] Lustre: *** cfs_fail_loc=159, val=0*** [ 3077.013500] Lustre: Skipped 6 previous similar messages [ 3084.427123] Lustre: lustre-MDT0000: Client fa2e8522-28e9-4ae4-aabb-a612e65c9a93 (at 192.168.203.42@tcp) reconnecting [ 3089.379658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.379696] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.380192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.397950] Lustre: Skipped 1 previous similar message [ 3089.412096] LustreError: Skipped 1 previous similar message [ 3093.007072] Lustre: server umount lustre-OST0000 complete [ 3093.010164] Lustre: Skipped 1 previous similar message [ 3099.104232] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3099.108586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.122170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.132208] Lustre: Skipped 1 previous similar message [ 3104.259913] LustreError: 133384:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940114 with bad export cookie 728172489606565545 [ 3104.262450] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.270945] LustreError: 133384:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3111.689070] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3113.213571] Key type lgssc unregistered [ 3113.363168] LNet: 134187:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3113.368199] LNet: Removed LNI 192.168.203.142@tcp [ 3113.873676] Key type .llcrypt unregistered [ 3113.875400] Key type ._llcrypt unregistered [ 3128.265506] alg: No test for adler32 (adler32-zlib) [ 3129.022581] Key type ._llcrypt registered [ 3129.024167] Key type .llcrypt registered [ 3129.096467] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3129.742820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3130.041640] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3130.188344] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3130.191442] LNet: Accept secure, port 988 [ 3131.799399] Key type lgssc registered [ 3132.551130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3137.088733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.370927] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.424085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.737826] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3143.649265] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.545557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.733488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3147.788611] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3151.418178] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3154.166497] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3157.731339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3157.919206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.215698] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3163.953798] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3169.196839] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3174.773587] Lustre: *** cfs_fail_loc=159, val=0*** [ 3182.191027] Lustre: lustre-MDT0000: Client 3ea03a6b-8fd0-42b1-9637-5e3466d08436 (at 192.168.203.42@tcp) reconnecting [ 3183.141695] Lustre: *** cfs_fail_loc=159, val=0*** [ 3190.382068] Lustre: lustre-MDT0001: Client 3ea03a6b-8fd0-42b1-9637-5e3466d08436 (at 192.168.203.42@tcp) reconnecting [ 3190.929117] Lustre: *** cfs_fail_loc=159, val=0*** [ 3190.932182] Lustre: Skipped 3 previous similar messages [ 3198.065874] Lustre: lustre-MDT0001: Client 3ea03a6b-8fd0-42b1-9637-5e3466d08436 (at 192.168.203.42@tcp) reconnecting [ 3201.268565] Lustre: *** cfs_fail_loc=159, val=0*** [ 3201.270729] Lustre: Skipped 3 previous similar messages [ 3207.773863] Lustre: lustre-MDT0000: Client b2901152-3a96-41c2-bb56-b5331e305720 (at 192.168.203.42@tcp) reconnecting [ 3208.373782] Lustre: *** cfs_fail_loc=159, val=0*** [ 3208.378162] Lustre: Skipped 13 previous similar messages [ 3215.465428] Lustre: lustre-MDT0000: Client b2901152-3a96-41c2-bb56-b5331e305720 (at 192.168.203.42@tcp) reconnecting [ 3220.449932] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3220.450374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.452913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3220.461117] Lustre: Skipped 1 previous similar message [ 3224.458248] Lustre: server umount lustre-OST0000 complete [ 3227.619074] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.622736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.627777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3227.630137] Lustre: Skipped 1 previous similar message [ 3230.688355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.688579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.697594] Lustre: Skipped 1 previous similar message [ 3232.792807] Lustre: server umount lustre-MDT0000 complete [ 3234.528498] LustreError: 135629:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940244 with bad export cookie 10766663002203716447 [ 3234.529917] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.533873] LustreError: 135629:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3234.673537] Lustre: server umount lustre-MDT0001 complete [ 3240.717032] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3241.902861] Key type lgssc unregistered [ 3242.026489] LNet: 140034:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.030058] LNet: Removed LNI 192.168.203.142@tcp [ 3242.387833] Key type .llcrypt unregistered [ 3242.389504] Key type ._llcrypt unregistered [ 3254.980659] alg: No test for adler32 (adler32-zlib) [ 3255.732464] Key type ._llcrypt registered [ 3255.734853] Key type .llcrypt registered [ 3255.783888] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3256.300220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3256.461034] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3256.578317] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3256.582082] LNet: Accept secure, port 988 [ 3258.175121] Key type lgssc registered [ 3258.714400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3263.064974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.321290] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.351212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.787748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3269.603164] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.945255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.234440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.542419] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3278.691165] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3282.352251] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.936152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.112912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.152389] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 3288.078942] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3291.317914] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3295.339488] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 3301.185841] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3303.050749] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3303.054452] Lustre: Skipped 1 previous similar message [ 3305.953842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.954443] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.955060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.959814] Lustre: Skipped 1 previous similar message [ 3311.072819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3311.080309] Lustre: Skipped 2 previous similar messages [ 3311.472840] Lustre: server umount lustre-OST0000 complete [ 3316.195755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.200231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.205189] Lustre: Skipped 1 previous similar message [ 3319.503749] Lustre: server umount lustre-MDT0000 complete [ 3321.082442] LustreError: 144160:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940331 with bad export cookie 7740403928148504238 [ 3321.084810] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.088208] LustreError: 144160:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3321.219164] Lustre: server umount lustre-MDT0001 complete [ 3327.200040] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3328.415630] Key type lgssc unregistered [ 3328.546471] LNet: 145550:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3328.549918] LNet: Removed LNI 192.168.203.142@tcp [ 3328.883412] Key type .llcrypt unregistered [ 3328.885481] Key type ._llcrypt unregistered [ 3340.173497] alg: No test for adler32 (adler32-zlib) [ 3340.926509] Key type ._llcrypt registered [ 3340.928092] Key type .llcrypt registered [ 3340.982051] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3341.495950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3341.721859] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3341.836987] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3341.840841] LNet: Accept secure, port 988 [ 3343.455123] Key type lgssc registered [ 3344.082932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3348.164766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.383327] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.414972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.554853] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3354.593514] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.654932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.848306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3357.633873] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3360.820974] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3363.503367] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3366.470758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3366.624599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.408983] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3370.728808] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 3371.968604] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3379.940843] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 3384.904094] Lustre: *** cfs_fail_loc=159, val=0*** [ 3384.908282] Lustre: Skipped 1 previous similar message [ 3392.115801] Lustre: lustre-MDT0000: Client c6b43a33-1430-4cea-bab1-1ab1f78aa12e (at 192.168.203.42@tcp) reconnecting [ 3395.556528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.557072] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.566127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3395.570199] Lustre: Skipped 1 previous similar message [ 3400.049661] Lustre: server umount lustre-OST0000 complete [ 3402.208470] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3402.212079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3402.217321] Lustre: Skipped 1 previous similar message [ 3402.219887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.792624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.792748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.952671] Lustre: server umount lustre-MDT0000 complete [ 3409.617835] LustreError: 149990:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940420 with bad export cookie 7462132026574760400 [ 3409.620291] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.623263] LustreError: 149990:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3409.764031] Lustre: server umount lustre-MDT0001 complete [ 3415.620294] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3416.856833] Key type lgssc unregistered [ 3416.988703] LNet: 150792:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3416.993222] LNet: Removed LNI 192.168.203.142@tcp [ 3417.357374] Key type .llcrypt unregistered [ 3417.359079] Key type ._llcrypt unregistered [ 3429.959131] alg: No test for adler32 (adler32-zlib) [ 3430.711500] Key type ._llcrypt registered [ 3430.713054] Key type .llcrypt registered [ 3430.781914] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3431.187079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3431.443221] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3431.548242] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3431.552242] LNet: Accept secure, port 988 [ 3433.151545] Key type lgssc registered [ 3433.616165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3437.130784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.341353] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.375478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.172352] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3443.682838] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.759742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.933152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3445.749755] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3449.236878] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3451.791416] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3454.700686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3454.851559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.631741] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3458.982223] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 3460.195224] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3469.226444] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 3484.489715] Lustre: lustre-OST0000: evicting 771790a0-51f9-4173-bce2-999ddb5ddcb3 (at 192.168.203.42@tcp) by administrative request [ 3484.494150] Lustre: Skipped 1 previous similar message [ 3510.239906] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3510.243413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.244251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.247590] Lustre: Skipped 1 previous similar message [ 3515.367599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3515.372686] Lustre: Skipped 2 previous similar messages [ 3515.617410] Lustre: server umount lustre-OST0000 complete [ 3520.480948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3520.482630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.486255] Lustre: Skipped 1 previous similar message [ 3523.477325] Lustre: server umount lustre-MDT0000 complete [ 3525.196418] LustreError: 154214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940535 with bad export cookie 8977821640423363006 [ 3525.199255] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3525.201226] LustreError: 154214:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3525.341635] Lustre: server umount lustre-MDT0001 complete [ 3531.115774] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3532.305243] Key type lgssc unregistered [ 3532.441550] LNet: 156467:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.445599] LNet: Removed LNI 192.168.203.142@tcp [ 3532.777955] Key type .llcrypt unregistered [ 3532.779814] Key type ._llcrypt unregistered [ 3565.659595] alg: No test for adler32 (adler32-zlib) [ 3566.411385] Key type ._llcrypt registered [ 3566.412846] Key type .llcrypt registered [ 3566.461954] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3566.827820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3567.128731] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3567.229032] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3567.232779] LNet: Accept secure, port 988 [ 3568.831301] Key type lgssc registered [ 3569.292103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3572.628188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.820213] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.845634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.579603] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3578.848426] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.368041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.554941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.203734] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3584.110780] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3586.469214] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3589.238739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3589.394351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.102308] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3594.297851] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3595.557977] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 3603.751231] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 3609.056652] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3609.057964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.067609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3609.069878] Lustre: Skipped 1 previous similar message [ 3614.170574] Lustre: server umount lustre-OST0000 complete [ 3615.711933] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3615.716057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.722826] Lustre: Skipped 1 previous similar message [ 3615.726105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3619.296330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3619.297440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.708930] Lustre: server umount lustre-MDT0000 complete [ 3623.202738] LustreError: 161677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940633 with bad export cookie 14640774950103857109 [ 3623.204332] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3623.208515] LustreError: 161677:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3623.336406] Lustre: server umount lustre-MDT0001 complete [ 3628.840907] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3629.946532] Key type lgssc unregistered [ 3630.079555] LNet: 162681:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3630.082381] LNet: Removed LNI 192.168.203.142@tcp [ 3630.392135] Key type .llcrypt unregistered [ 3630.393555] Key type ._llcrypt unregistered [ 3640.611096] alg: No test for adler32 (adler32-zlib) [ 3641.363473] Key type ._llcrypt registered [ 3641.364853] Key type .llcrypt registered [ 3641.421776] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3641.924542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3642.145752] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3642.252697] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3642.254866] LNet: Accept secure, port 988 [ 3643.855190] Key type lgssc registered [ 3644.330652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3647.383129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3650.401294] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 08:17:40 (1764940660) [ 3656.730520] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 3660.654628] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3664.902378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3667.792526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3670.392926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3673.822137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3677.324291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3677.361429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.485048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3678.510632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3678.559570] Lustre: lustre-MDT0000: new disk, initializing [ 3678.609340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.628468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3680.758578] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3685.841293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3685.874356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.917207] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3685.945520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3685.948831] Lustre: Skipped 1 previous similar message [ 3685.995472] Lustre: lustre-MDT0001: new disk, initializing [ 3686.044238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.063729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3686.067871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3688.015862] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3692.621988] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3695.039629] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3697.855444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3697.899401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3697.903302] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3698.043353] Lustre: lustre-OST0000: new disk, initializing [ 3698.047237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3698.101990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.077115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3699.081239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3699.916163] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3704.773453] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3709.411605] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.411808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.412644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3709.422663] Lustre: Skipped 1 previous similar message [ 3714.529242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3714.532637] Lustre: Skipped 2 previous similar messages [ 3714.992893] Lustre: server umount lustre-OST0000 complete [ 3717.087953] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3717.091460] LustreError: Skipped 1 previous similar message [ 3717.093558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.099083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3719.647992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3719.648556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.766533] Lustre: server umount lustre-MDT0000 complete [ 3724.407477] LustreError: 169163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940734 with bad export cookie 1395352558527467536 [ 3724.411837] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.412868] LustreError: 169163:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3724.545132] Lustre: server umount lustre-MDT0001 complete [ 3729.809585] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3731.007105] Key type lgssc unregistered [ 3731.141615] LNet: 171076:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3731.146304] LNet: Removed LNI 192.168.203.142@tcp [ 3731.543319] Key type .llcrypt unregistered [ 3731.545381] Key type ._llcrypt unregistered [ 3742.317434] alg: No test for adler32 (adler32-zlib) [ 3743.069345] Key type ._llcrypt registered [ 3743.070946] Key type .llcrypt registered [ 3743.120878] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3743.572975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3743.786849] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3743.886087] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3743.889137] LNet: Accept secure, port 988 [ 3745.487296] Key type lgssc registered [ 3746.007267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3749.729033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.945551] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3750.973277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.845632] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3756.001189] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.643287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.817225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3758.498816] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3761.637588] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3764.036662] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3766.810832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3766.980788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3768.851912] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3772.156362] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3781.351747] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 3784.899979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3784.909996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.916068] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3784.920179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3784.925077] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 3784.926277] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 3785.328620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3785.334826] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 3785.724691] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3785.728312] Lustre: Skipped 1 previous similar message [ 3785.734922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.741882] Lustre: Skipped 1 previous similar message [ 3785.744600] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3785.748097] Lustre: Skipped 1 previous similar message [ 3785.749933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3785.754448] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 3785.754746] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:161 [ 3785.757389] Lustre: Skipped 1 previous similar message [ 3786.138453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3786.144171] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:193 [ 3786.515849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3786.521387] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:225 [ 3786.929214] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3786.931721] Lustre: Skipped 2 previous similar messages [ 3786.938738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.944292] Lustre: Skipped 2 previous similar messages [ 3786.946400] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3786.949676] Lustre: Skipped 2 previous similar messages [ 3786.951507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3786.956614] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 3786.956922] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:257 [ 3786.960080] Lustre: Skipped 2 previous similar messages [ 3787.408566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3787.415588] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:289 [ 3787.909432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3787.914352] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:321 [ 3788.325154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3788.330153] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:353 [ 3788.687159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3788.691212] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:385 [ 3791.842639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.843792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.848926] Lustre: Skipped 5 previous similar messages [ 3791.853936] Lustre: Skipped 1 previous similar message [ 3796.961351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3796.963926] Lustre: Skipped 1 previous similar message [ 3802.080211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.082481] Lustre: Skipped 1 previous similar message [ 3806.175140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3806.249696] Lustre: server umount lustre-OST0000 complete [ 3808.224065] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3808.227616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.234363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.320378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.322697] Lustre: Skipped 1 previous similar message [ 3813.961474] Lustre: server umount lustre-MDT0000 complete [ 3815.544900] LustreError: 172520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940825 with bad export cookie 13087227356212124506 [ 3815.548320] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.552087] LustreError: 172520:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3815.718925] Lustre: server umount lustre-MDT0001 complete [ 3821.373780] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3822.657367] Key type lgssc unregistered [ 3822.780765] LNet: 177013:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3822.784298] LNet: Removed LNI 192.168.203.142@tcp [ 3823.134176] Key type .llcrypt unregistered [ 3823.135807] Key type ._llcrypt unregistered [ 3833.197477] alg: No test for adler32 (adler32-zlib) [ 3833.949458] Key type ._llcrypt registered [ 3833.951246] Key type .llcrypt registered [ 3833.994839] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3834.353791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3834.625172] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3834.724345] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3834.727688] LNet: Accept secure, port 988 [ 3836.327098] Key type lgssc registered [ 3836.760345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3839.830117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.003572] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3841.020075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.608510] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3845.973733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.148851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3847.872771] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3850.831193] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3853.012178] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3855.468283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3855.594069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.617195] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:417 [ 3857.259836] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3860.090360] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3863.036121] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3863.703596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.709175] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3863.713046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3863.718799] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 3863.718820] LustreError: 178495:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 3866.084989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.085159] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3866.092079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3866.096899] Lustre: Skipped 1 previous similar message [ 3866.726398] Lustre: lustre-OST0000: Not available for connect from 192.168.203.42@tcp (stopping) [ 3870.555589] Lustre: server umount lustre-OST0000 complete [ 3883.799124] LustreError: 178495:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 3883.803313] LustreError: 178495:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3899.167497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.312086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.927681] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3903.899245] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3904.557487] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3906.949888] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91bd8aabc800.ost_server_uuid 40 [ 3907.575289] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91bd8aabc800.ost_server_uuid in FULL state after 0 sec [ 3918.692624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3918.697913] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3918.697937] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.142@tcp (at 0@lo) [ 3918.700293] Lustre: lustre-OST0000: deleting orphan objects from 0x0:146 to 0x0:449 [ 3918.706857] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 3918.707109] Lustre: Skipped 1 previous similar message [ 3923.937190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.940582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.946195] Lustre: Skipped 2 previous similar messages [ 3923.951250] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.113223] Lustre: server umount lustre-OST0000 complete [ 3928.031907] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3928.034955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.039146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.041149] Lustre: Skipped 1 previous similar message [ 3934.175992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3934.179348] Lustre: Skipped 3 previous similar messages [ 3942.367127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3942.410903] Lustre: server umount lustre-MDT0000 complete [ 3943.849324] LustreError: 182994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764940954 with bad export cookie 17537497127365597961 [ 3943.851689] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3943.853920] LustreError: 182994:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3949.345486] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 3950.610613] Key type lgssc unregistered [ 3950.738799] LNet: 183797:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3950.743455] LNet: Removed LNI 192.168.203.142@tcp [ 3951.122450] Key type .llcrypt unregistered [ 3951.124125] Key type ._llcrypt unregistered [ 3964.016416] alg: No test for adler32 (adler32-zlib) [ 3964.767668] Key type ._llcrypt registered [ 3964.769263] Key type .llcrypt registered [ 3964.814714] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 3965.285964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.454732] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 3965.562303] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 3965.564499] LNet: Accept secure, port 988 [ 3967.151121] Key type lgssc registered [ 3967.590688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3971.153274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.300734] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.317550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.926283] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3977.110671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.228900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3978.791724] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3981.933281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3982.067231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.730946] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3987.276340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3987.311480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.315854] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 3987.393436] Lustre: lustre-OST0001: new disk, initializing [ 3987.397605] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 3987.449026] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3988.523748] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 3988.527472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3988.532062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3989.190852] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 3994.809245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3997.645000] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4013.023914] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4013.027317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.033112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.051832] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.052850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.057021] Lustre: Skipped 2 previous similar messages [ 4014.061709] Lustre: Skipped 3 previous similar messages [ 4017.430240] Lustre: server umount lustre-MDT0000 complete [ 4018.927140] LustreError: 185554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941029 with bad export cookie 17034144465680394132 [ 4018.930571] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4018.932094] LustreError: 185554:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4019.065641] Lustre: server umount lustre-MDT0001 complete [ 4026.271153] Lustre: 184838:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941029/real 1764941029] req@00000000c93c05a7 x1850674747843072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764941036 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:5.0' [ 4026.282395] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4026.847135] Lustre: 189363:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941031/real 1764941031] req@00000000abd77115 x1850674747843200/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764941037 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4026.869946] Lustre: server umount lustre-OST0000 complete [ 4029.751070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4029.793183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.261155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4030.293191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.773675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4030.814333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4031.296866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4031.343830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4036.705318] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4040.452361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.634156] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.660732] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.308121] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4045.471179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4047.220963] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4050.215305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.311108] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4050.318727] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4050.321387] Lustre: Skipped 1 previous similar message [ 4051.854812] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4054.877154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.464485] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4057.127839] Lustre: mylustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 4057.129041] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 4059.713175] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4060.884927] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4071.392087] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4071.395954] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.402084] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4076.360209] Lustre: server umount mylustre-MDT0000 complete [ 4076.361949] Lustre: Skipped 1 previous similar message [ 4077.536621] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.542683] LustreError: Skipped 3 previous similar messages [ 4077.687125] LustreError: 191003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941088 with bad export cookie 17034144465680396841 [ 4077.688967] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4077.689782] LustreError: 191003:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4085.215124] Lustre: 194720:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941089/real 1764941089] req@0000000070e57399 x1850674747866624/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764941095 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4085.235397] Lustre: server umount mylustre-OST0000 complete [ 4085.236652] Lustre: Skipped 1 previous similar message [ 4092.895207] Lustre: 194921:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941097/real 1764941097] req@00000000a5352473 x1850674747867008/t0(0) o39->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764941103 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4094.118240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4094.150668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.513506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4094.543176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4094.926379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4094.954395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4095.334902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4095.371322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4100.353795] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4103.728805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.874737] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.899709] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.902587] Lustre: Skipped 1 previous similar message [ 4105.407718] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4108.507846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.314872] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4113.374157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4114.529863] LustreError: 137-5: tfs-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.033556] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4119.324980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.423573] Lustre: tfs-OST0000: deleting orphan objects from 0x0:483 to 0x0:545 [ 4120.423669] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 4120.953084] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4124.267111] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4125.527554] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4134.367941] LustreError: 11-0: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4134.370697] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.374511] Lustre: Skipped 3 previous similar messages [ 4134.376273] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 4134.378795] Lustre: Skipped 4 previous similar messages [ 4138.502887] Lustre: server umount tfs-MDT0000 complete [ 4138.504325] Lustre: Skipped 1 previous similar message [ 4139.735076] LustreError: 196363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941150 with bad export cookie 17034144465680398668 [ 4139.738255] LustreError: 196363:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4139.738717] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4146.655121] Lustre: 200023:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941151/real 1764941151] req@000000007735c23a x1850674747885696/t0(0) o39->tfs-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764941157 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4155.563700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4155.594793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4155.990896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4156.023633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4156.407574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4156.435643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4156.863847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4156.908492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4162.007471] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4165.491118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4165.640076] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.658649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.660572] Lustre: Skipped 3 previous similar messages [ 4167.221203] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4170.213164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.875328] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4175.084632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.758635] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4179.735805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4180.840941] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 4180.841264] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:577 [ 4181.289542] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4184.531305] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4185.753100] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4192.903046] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 08:26:42 (1764941202) [ 4206.047760] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4206.050121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.053126] Lustre: Skipped 3 previous similar messages [ 4206.054555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4206.056228] Lustre: Skipped 4 previous similar messages [ 4210.760460] Lustre: server umount lustre-MDT0000 complete [ 4210.762442] Lustre: Skipped 3 previous similar messages [ 4211.680850] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.685406] LustreError: Skipped 4 previous similar messages [ 4211.975152] LustreError: 201665:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941222 with bad export cookie 17034144465680400306 [ 4211.977349] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.979567] LustreError: 201665:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4216.801188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4216.803674] Lustre: Skipped 4 previous similar messages [ 4241.375129] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4243.567698] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4246.711539] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4250.173332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4252.568073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4254.688835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4256.934996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4259.387052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4259.409266] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4259.409266] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4259.409266] [ 4259.416541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4259.518292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4259.530105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4259.569210] Lustre: lustre-MDT0000: new disk, initializing [ 4259.595932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.599995] Lustre: Skipped 3 previous similar messages [ 4259.608750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4261.212449] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4265.743309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4265.768626] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 4265.768626] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4265.768626] [ 4265.779154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4265.833692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4265.836086] Lustre: Skipped 1 previous similar message [ 4265.875347] Lustre: lustre-MDT0001: new disk, initializing [ 4265.923700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4265.928969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4267.500658] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4271.964831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4271.991849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4271.994964] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4272.086424] Lustre: lustre-OST0000: new disk, initializing [ 4272.088840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4273.679222] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4273.787798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4273.792456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4277.925605] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4277.951123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4277.954494] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4278.001602] Lustre: lustre-OST0001: new disk, initializing [ 4278.003188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4279.441882] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4279.805934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4279.812923] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4283.773827] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 08:28:13 (1764941293) [ 4287.364573] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 08:28:17 (1764941297) [ 4299.744186] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.746685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.749518] Lustre: Skipped 5 previous similar messages [ 4299.751044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.753705] Lustre: Skipped 3 previous similar messages [ 4313.055110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4313.070323] Lustre: server umount lustre-OST0000 complete [ 4313.072253] Lustre: Skipped 3 previous similar messages [ 4315.617232] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.621915] LustreError: Skipped 4 previous similar messages [ 4321.789267] LustreError: 207937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941332 with bad export cookie 17034144465680401825 [ 4321.791980] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4321.792669] LustreError: 207937:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4333.024052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4333.026461] Lustre: Skipped 12 previous similar messages [ 4336.095108] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4340.788369] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4352.913913] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4358.722401] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4359.529578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4362.175074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.465829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.525425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4368.687334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4371.128571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4371.154729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.247212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4371.250539] Lustre: Skipped 1 previous similar message [ 4371.261875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4371.303971] Lustre: lustre-MDT0000: new disk, initializing [ 4371.337982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4372.879667] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4377.282423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4377.309269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.441487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4379.022727] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4382.881225] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4384.861965] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4387.240361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4387.263811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.266860] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4387.360176] Lustre: lustre-OST0000: new disk, initializing [ 4387.361342] Lustre: Skipped 1 previous similar message [ 4387.363503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4387.366245] Lustre: Skipped 2 previous similar messages [ 4388.889405] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4389.362938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4389.366546] Lustre: Skipped 1 previous similar message [ 4389.368398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4392.831220] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4399.585135] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4399.586640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.588433] LustreError: Skipped 2 previous similar messages [ 4399.593074] Lustre: Skipped 5 previous similar messages [ 4399.593769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4399.599871] Lustre: Skipped 1 previous similar message [ 4401.236399] Lustre: server umount lustre-OST0000 complete [ 4401.237685] Lustre: Skipped 3 previous similar messages [ 4409.893939] LustreError: 215119:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941420 with bad export cookie 17034144465680403253 [ 4409.896087] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.897475] LustreError: 215119:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4414.739447] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4415.753108] Key type lgssc unregistered [ 4415.875342] LNet: 218977:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4415.877664] LNet: Removed LNI 192.168.203.142@tcp [ 4416.151372] Key type .llcrypt unregistered [ 4416.152864] Key type ._llcrypt unregistered [ 4425.707387] alg: No test for adler32 (adler32-zlib) [ 4426.460518] Key type ._llcrypt registered [ 4426.462427] Key type .llcrypt registered [ 4426.502130] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4426.925128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4427.253926] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 4427.341698] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4427.343575] LNet: Accept secure, port 988 [ 4428.927161] Key type lgssc registered [ 4429.301715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4432.102365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.239359] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.253664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.732397] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4437.704454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.830443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.342709] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4441.819557] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4443.716650] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.049899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.173047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.685548] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4449.254493] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4450.246605] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4450.601673] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 4453.346289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.346987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.368528] Lustre: server umount lustre-OST0000 complete [ 4463.584751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.585172] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4463.585748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.585754] Lustre: Skipped 1 previous similar message [ 4463.590075] Lustre: Skipped 1 previous similar message [ 4464.839200] Lustre: server umount lustre-MDT0000 complete [ 4466.096951] LustreError: 220488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941476 with bad export cookie 10856669778755173894 [ 4466.099140] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4466.100640] LustreError: 220488:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4466.217281] Lustre: server umount lustre-MDT0001 complete [ 4469.650934] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4470.583114] Key type lgssc unregistered [ 4470.697314] LNet: 224213:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4470.699567] LNet: Removed LNI 192.168.203.142@tcp [ 4470.967796] Key type .llcrypt unregistered [ 4470.968650] Key type ._llcrypt unregistered [ 4476.784739] alg: No test for adler32 (adler32-zlib) [ 4477.536329] Key type ._llcrypt registered [ 4477.537140] Key type .llcrypt registered [ 4477.575827] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4477.925263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4478.191180] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 4478.264197] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4478.265665] LNet: Accept secure, port 988 [ 4479.839138] Key type lgssc registered [ 4480.190755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4483.034201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.511336] LustreError: 225453:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 1: rc = -5 [ 4531.231186] LustreError: 225453:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.203.142@tcp: failed processing log, type 4: rc = -110 [ 4561.951955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.570555] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4573.464971] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4575.550934] Lustre: server umount lustre-OST0000 complete [ 4596.133730] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4599.881543] Key type lgssc unregistered [ 4600.288420] LNet: 227099:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4600.311588] LNet: Removed LNI 192.168.203.142@tcp [ 4601.359165] Key type .llcrypt unregistered [ 4601.363972] Key type ._llcrypt unregistered [ 4624.231740] alg: No test for adler32 (adler32-zlib) [ 4624.988839] Key type ._llcrypt registered [ 4624.990804] Key type .llcrypt registered [ 4625.042702] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4625.496254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4625.835175] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 4625.998411] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 4626.002640] LNet: Accept secure, port 988 [ 4627.607117] Key type lgssc registered [ 4628.173225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4634.116943] loop0: detected capacity change from 0 to 204800000 [ 4634.217298] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.222589] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.384983] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.390897] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.423357] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.451916] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.477109] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.515300] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.555776] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4634.599349] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4635.128486] loop0: detected capacity change from 0 to 204800000 [ 4635.229723] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4635.590619] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4636.102760] loop0: detected capacity change from 0 to 204800000 [ 4636.315513] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4636.936115] loop0: detected capacity change from 0 to 204800000 [ 4637.190306] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4637.806164] loop0: detected capacity change from 0 to 204800000 [ 4637.832484] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4639.923079] loop0: detected capacity change from 0 to 204800000 [ 4639.938568] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4641.526433] loop0: detected capacity change from 0 to 204800000 [ 4641.541825] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4643.299298] loop0: detected capacity change from 0 to 204800000 [ 4643.320611] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 4645.346277] loop0: detected capacity change from 0 to 204800000 [ 4645.401358] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.451524] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.486915] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4645.492799] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4645.498694] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4645.504285] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4645.512627] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 4645.520897] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 4645.529283] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 4645.536108] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 4645.544591] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 4645.550675] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 4645.556784] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 4645.562794] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 4645.569253] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 4645.576430] Lustre: 230556:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 4655.583180] Lustre: 230834:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941659/real 1764941659] req@000000007a01ee9b x1850675440910464/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764941665 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4655.612344] Lustre: server umount MGS complete [ 4656.160949] loop0: detected capacity change from 0 to 204800000 [ 4656.217092] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4656.484891] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4656.529764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.558365] LustreError: 230888:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 4657.139419] loop1: detected capacity change from 0 to 204800000 [ 4657.201525] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4657.237189] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.280214] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4657.288805] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 4657.299864] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4657.310175] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4657.317957] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 4657.324676] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4657.329813] Lustre: 231004:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 4657.345328] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4657.473345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4657.507361] LustreError: 231004:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 4658.020187] loop2: detected capacity change from 0 to 204800000 [ 4658.068081] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.136102] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.141189] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 4658.170291] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4658.175237] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 4658.181213] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 4658.188581] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 4658.193703] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4658.199787] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 4658.204979] Lustre: 231088:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 4658.280910] Lustre: Found index 0 for lustre-OST0000, updating log [ 4658.357319] LustreError: 230946:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 4658.366335] LustreError: 231088:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 4658.917370] loop3: detected capacity change from 0 to 204800000 [ 4659.012144] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4659.076831] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.082920] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 4659.119466] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 4659.125732] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 4659.133288] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 4659.144554] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 4659.153349] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 4659.160240] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 4659.167974] Lustre: 231174:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 4659.251931] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4659.254707] Lustre: Skipped 1 previous similar message [ 4659.268604] LustreError: 231174:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 4661.289875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 4661.294169] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 4662.308050] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 4662.308189] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 4665.768117] LustreError: 230911:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1219us: rc = -2 [ 4666.273619] LustreError: 230911:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2015us: rc = -2 [ 4666.289849] LustreError: 230911:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 40 previous similar messages [ 4677.600911] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.601606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4677.612654] Lustre: Skipped 3 previous similar messages [ 4677.615570] Lustre: Skipped 2 previous similar messages [ 4682.722390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4682.725378] Lustre: Skipped 4 previous similar messages [ 4683.536494] Lustre: server umount lustre-MDT0000 complete [ 4684.092385] LustreError: 230895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941694 with bad export cookie 13340661160165697359 [ 4684.097319] LustreError: 230895:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4684.097950] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.840348] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.841352] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.841940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4687.845509] Lustre: Skipped 1 previous similar message [ 4687.855227] LustreError: Skipped 1 previous similar message [ 4689.887809] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4689.887961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4689.894150] LustreError: Skipped 1 previous similar message [ 4689.896467] Lustre: Skipped 2 previous similar messages [ 4690.322433] Lustre: server umount lustre-MDT0001 complete [ 4705.247216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4705.263528] Lustre: server umount lustre-OST0000 complete [ 4711.801187] Lustre: server umount lustre-OST0001 complete [ 4716.605910] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 08:35:26 (1764941726) [ 4729.280641] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 4733.709793] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 4738.323754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4741.495485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4744.397651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4748.124775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4752.364314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4752.432605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4752.600473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4752.627521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4752.669446] Lustre: lustre-MDT0000: new disk, initializing [ 4752.725149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.736575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4755.620886] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4762.794260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4762.858460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.928571] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4762.961850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4762.964731] Lustre: Skipped 1 previous similar message [ 4763.008992] Lustre: lustre-MDT0001: new disk, initializing [ 4763.070522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4763.095890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4763.103660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4766.084524] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4772.497226] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4776.322838] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4780.330710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4780.390828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4780.398836] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4780.533677] Lustre: lustre-OST0000: new disk, initializing [ 4780.536421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4780.583220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.994865] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4788.756403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4788.762384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4789.775295] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4793.669433] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 4800.187814] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 4805.500189] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 4855.265726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4855.270545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4855.276145] Lustre: Skipped 1 previous similar message [ 4855.292866] Lustre: Skipped 1 previous similar message [ 4856.356365] Lustre: server umount lustre-OST0000 complete [ 4860.385061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.385282] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4865.506856] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.517376] LustreError: Skipped 1 previous similar message [ 4867.331657] LustreError: 237429:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941877 with bad export cookie 13340661160165700026 [ 4867.332691] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4867.342276] LustreError: 237429:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4872.042941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4874.221748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4883.427416] Lustre: 240731:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764941887/real 1764941887] req@000000006082c11b x1850675440974080/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764941893 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4883.462110] Lustre: server umount MGS complete [ 4883.463807] Lustre: Skipped 2 previous similar messages [ 4889.404335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.662110] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.676252] LustreError: 241366:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 4889.714150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4892.167881] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4897.100454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4899.728898] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4904.423849] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4908.420628] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4912.589814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4915.663910] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 4920.834029] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4942.817200] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4942.818620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.838334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4942.845653] Lustre: Skipped 2 previous similar messages [ 4942.850130] Lustre: Skipped 3 previous similar messages [ 4948.892038] Lustre: server umount lustre-OST0000 complete [ 4953.056697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.067896] Lustre: Skipped 1 previous similar message [ 4958.177640] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.185572] LustreError: Skipped 2 previous similar messages [ 4959.561061] LustreError: 244149:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764941969 with bad export cookie 13340661160165701825 [ 4959.565817] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4959.580937] LustreError: 244149:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4967.089886] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 4968.741483] Key type lgssc unregistered [ 4968.927523] LNet: 245467:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4968.933478] LNet: Removed LNI 192.168.203.142@tcp [ 4969.391472] Key type .llcrypt unregistered [ 4969.393382] Key type ._llcrypt unregistered [ 4991.766137] alg: No test for adler32 (adler32-zlib) [ 4992.519505] Key type ._llcrypt registered [ 4992.520920] Key type .llcrypt registered [ 4992.605047] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5001.207366] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5001.626553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5001.930092] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 5002.032252] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5002.035278] LNet: Accept secure, port 988 [ 5003.631241] Key type lgssc registered [ 5004.097555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5008.784970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.491942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5015.797516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5019.358103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5023.467887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5023.508629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.622428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5024.635293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5024.676761] Lustre: lustre-MDT0000: new disk, initializing [ 5024.720889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.731864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5026.997420] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5032.854302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5032.897853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.953163] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5032.977640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5032.981558] Lustre: Skipped 1 previous similar message [ 5033.029188] Lustre: lustre-MDT0001: new disk, initializing [ 5033.078690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.097315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5033.107235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5035.178257] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5040.851992] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5043.786499] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5047.672846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5047.710544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5047.717540] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5047.860617] Lustre: lustre-OST0000: new disk, initializing [ 5047.873738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5047.922372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.247979] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5053.448856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5053.454264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5055.931795] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5060.185881] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5063.465451] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5070.267112] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5114.850670] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.851210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.858295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.862961] Lustre: Skipped 1 previous similar message [ 5114.868415] Lustre: Skipped 1 previous similar message [ 5119.972206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.977330] Lustre: Skipped 1 previous similar message [ 5125.092052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5125.094885] Lustre: Skipped 1 previous similar message [ 5127.135716] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5127.220293] Lustre: server umount lustre-OST0000 complete [ 5130.208602] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5130.209591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.212895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5130.217989] Lustre: Skipped 1 previous similar message [ 5130.222189] Lustre: Skipped 1 previous similar message [ 5134.983658] Lustre: server umount lustre-MDT0000 complete [ 5135.328186] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.332744] LustreError: Skipped 1 previous similar message [ 5136.450679] LustreError: 249536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764942146 with bad export cookie 11925264178322408674 [ 5136.453565] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.456282] LustreError: 249536:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5136.599105] Lustre: server umount lustre-MDT0001 complete [ 5140.348752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.212664] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5149.663151] Lustre: 254881:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764942154/real 1764942154] req@00000000ffe08a49 x1850675835217728/t0(0) o251->MGC192.168.203.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764942160 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5149.676619] Lustre: server umount MGS complete [ 5154.334826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.531228] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.543419] LustreError: 255516:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5154.571293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.311251] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5159.905184] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.069220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.271641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5162.105371] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5165.402949] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5168.480726] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5171.679979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.806713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.563249] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5176.781297] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5211.106872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.107321] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5211.113109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5211.115949] LustreError: Skipped 1 previous similar message [ 5211.118133] Lustre: Skipped 1 previous similar message [ 5213.411615] Lustre: server umount lustre-OST0000 complete [ 5216.224663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.230768] Lustre: Skipped 2 previous similar messages [ 5221.348383] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.353582] LustreError: Skipped 1 previous similar message [ 5222.947039] LustreError: 255484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764942233 with bad export cookie 11925264178322410459 [ 5222.950245] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5222.952495] LustreError: 255484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5223.110318] Lustre: server umount lustre-MDT0001 complete [ 5223.111900] Lustre: Skipped 1 previous similar message [ 5228.920936] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5230.058563] Key type lgssc unregistered [ 5230.169866] LNet: 259639:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5230.173965] LNet: Removed LNI 192.168.203.142@tcp [ 5230.493662] Key type .llcrypt unregistered [ 5230.495356] Key type ._llcrypt unregistered [ 5242.186540] alg: No test for adler32 (adler32-zlib) [ 5242.938370] Key type ._llcrypt registered [ 5242.939710] Key type .llcrypt registered [ 5242.991809] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5243.499800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5243.650080] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 5243.746629] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5243.749975] LNet: Accept secure, port 988 [ 5245.343129] Key type lgssc registered [ 5245.806241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5249.086050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.265669] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5250.294206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.017955] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5255.339598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.494081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5257.120195] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5259.970203] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5262.288239] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5264.878985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.040644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.861208] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5270.036064] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5273.112902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5273.148038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5273.151621] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 5273.227266] Lustre: lustre-OST0001: new disk, initializing [ 5273.229910] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 5273.289397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5273.346549] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5274.401744] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5275.233615] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5275.426818] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5277.473057] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5277.478053] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 5278.695057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5278.699815] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5281.013410] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5281.103664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5281.569200] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 5281.574170] LustreError: 264130:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [ 5283.375428] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91bd841ea800.ost_server_uuid 40 [ 5283.949094] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91bd841ea800.ost_server_uuid in FULL state after 0 sec [ 5286.206218] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91bd841ea800.ost_server_uuid 40 [ 5286.790666] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91bd841ea800.ost_server_uuid in FULL state after 0 sec [ 5299.813549] Lustre: lustre-OST0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 5300.194116] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.926209] Lustre: lustre-OST0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 5304.929697] Lustre: Skipped 2 previous similar messages [ 5310.045294] Lustre: lustre-OST0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 5310.048604] Lustre: Skipped 2 previous similar messages [ 5312.991130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5313.007992] Lustre: server umount lustre-OST0001 complete [ 5315.558211] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.559749] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.560478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.560490] Lustre: Skipped 1 previous similar message [ 5315.561632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5315.561637] Lustre: Skipped 2 previous similar messages [ 5315.563635] LustreError: Skipped 1 previous similar message [ 5320.672939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.675382] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.676612] Lustre: Skipped 2 previous similar messages [ 5320.683898] LustreError: Skipped 1 previous similar message [ 5321.064681] Lustre: server umount lustre-OST0000 complete [ 5325.793138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5325.798928] Lustre: Skipped 2 previous similar messages [ 5328.716204] Lustre: server umount lustre-MDT0000 complete [ 5330.223405] LustreError: 261186:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764942340 with bad export cookie 12529476676083414592 [ 5330.225404] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.228697] LustreError: 261186:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5336.108988] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5337.309282] Key type lgssc unregistered [ 5337.452551] LNet: 266854:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5337.456614] LNet: Removed LNI 192.168.203.142@tcp [ 5337.870631] Key type .llcrypt unregistered [ 5337.872274] Key type ._llcrypt unregistered [ 5355.667216] alg: No test for adler32 (adler32-zlib) [ 5356.419379] Key type ._llcrypt registered [ 5356.421397] Key type .llcrypt registered [ 5356.476723] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5356.850427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5357.111912] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 5357.215513] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5357.221321] LNet: Accept secure, port 988 [ 5358.831139] Key type lgssc registered [ 5359.336476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5362.828348] loop0: detected capacity change from 0 to 1099511627776 [ 5363.028416] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 5363.032250] *** VALIDATE xfs *** [ 5363.039701] XFS (loop0): Mounting V5 Filesystem [ 5363.056073] XFS (loop0): Ending clean mount [ 5363.652139] loop1: detected capacity change from 0 to 18691697672192 [ 5387.983917] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5396.214776] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 5400.149302] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5404.382879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5407.315978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.048217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5412.551779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.275622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5415.304261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.399565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5416.409431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5416.447552] Lustre: lustre-MDT0000: new disk, initializing [ 5416.476893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.484802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5418.219541] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5423.114232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5423.152959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.203507] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5423.230471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5423.233324] Lustre: Skipped 1 previous similar message [ 5423.277525] Lustre: lustre-MDT0001: new disk, initializing [ 5423.318323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5423.333771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5423.338314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5425.163106] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5430.102717] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5432.492490] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5435.398644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5435.448232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5435.453194] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5435.563989] Lustre: lustre-OST0000: new disk, initializing [ 5435.566414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5435.597430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.337960] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5441.543060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5441.548062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5442.105390] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5444.096950] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 5444.199390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5446.263833] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 5446.358993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5451.745544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.746594] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5451.746717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.751785] Lustre: Skipped 1 previous similar message [ 5453.419576] Lustre: server umount lustre-OST0000 complete [ 5456.864745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5456.865725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5456.870466] Lustre: Skipped 1 previous similar message [ 5456.873315] Lustre: Skipped 1 previous similar message [ 5461.447290] Lustre: server umount lustre-MDT0000 complete [ 5461.984866] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.991031] LustreError: Skipped 1 previous similar message [ 5462.981299] LustreError: 271785:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764942473 with bad export cookie 13727959696559293536 [ 5462.983319] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5462.986836] LustreError: 271785:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5463.110507] Lustre: server umount lustre-MDT0001 complete [ 5466.921376] XFS (loop0): Unmounting Filesystem [ 5468.921587] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 08:47:58 (1764942478) [ 5471.707278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.876601] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5471.911156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5473.639001] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5477.202350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.379937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5479.146656] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5482.041461] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5484.203856] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5486.910196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5487.049174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.861717] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5491.915089] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5497.312232] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5497.313340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5497.316200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5497.320534] Lustre: Skipped 1 previous similar message [ 5497.325031] Lustre: Skipped 2 previous similar messages [ 5502.372327] Lustre: server umount lustre-OST0000 complete [ 5507.552485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.553150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.557609] Lustre: Skipped 1 previous similar message [ 5507.562447] Lustre: Skipped 1 previous similar message [ 5510.221800] Lustre: server umount lustre-MDT0000 complete [ 5511.798214] LustreError: 276466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764942522 with bad export cookie 13727959696559294376 [ 5511.798872] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5511.803204] LustreError: 276466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5517.461333] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 5518.650867] Key type lgssc unregistered [ 5518.781414] LNet: 280231:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5518.785679] LNet: Removed LNI 192.168.203.142@tcp [ 5519.130122] Key type .llcrypt unregistered [ 5519.131716] Key type ._llcrypt unregistered [ 5530.821703] alg: No test for adler32 (adler32-zlib) [ 5531.573389] Key type ._llcrypt registered [ 5531.574917] Key type .llcrypt registered [ 5531.644160] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 5532.184736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5532.346260] Lustre: Lustre: Build Version: 2.15.8_1_gd7491a2 [ 5532.454424] LNet: Added LNI 192.168.203.142@tcp [8/256/0/180] [ 5532.456948] LNet: Accept secure, port 988 [ 5534.055121] Key type lgssc registered [ 5534.491374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5537.915907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.102583] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.129546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.876914] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5544.169732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.307327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5545.928302] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5548.740405] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5550.863339] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5553.376906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5553.514645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5555.079801] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5558.049778] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5563.873472] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.880278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5563.883308] Lustre: Skipped 1 previous similar message [ 5564.523949] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 5568.022119] Lustre: server umount lustre-MDT0001 complete [ 5568.993077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.999268] LustreError: Skipped 1 previous similar message [ 5569.438476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5572.840136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.890420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5572.892547] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5572.909704] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5573.034842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5574.840581] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5578.211757] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5578.218487] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5578.728644] LustreError: 285635:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 5578.733112] LustreError: 285635:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5578.736927] LustreError: 285635:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5578.741334] LustreError: 281838:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5578.768747] LustreError: 285638:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -17 [ 5578.773501] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 5578.773501] [ 5880.288549] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.289049] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5880.292943] Lustre: Skipped 1 previous similar message [ 5880.296274] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5880.298247] Lustre: Skipped 1 previous similar message [ 5883.968591] Lustre: server umount lustre-MDT0001 complete [ 5884.933193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5885.408829] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5885.411853] LustreError: Skipped 2 previous similar messages [ 5887.418706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.455414] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5887.457534] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5887.471311] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5887.568344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5888.969797] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 5892.579847] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5892.588373] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5892.592237] Lustre: Skipped 2 previous similar messages [ 5894.629650] LustreError: 298023:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 5894.633405] LustreError: 298023:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 5894.636959] LustreError: 298023:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5894.640035] LustreError: 298023:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5894.644578] LustreError: 281838:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5894.647132] LustreError: 281838:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5894.664521] LustreError: 298026:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 5894.667603] LustreError: 298026:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -17 [ 5894.671894] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.142@tcp [ 5894.671894] [ 6194.274525] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.42@tcp (stopping) [ 6194.276902] Lustre: Skipped 1 previous similar message [ 6194.657559] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.661522] Lustre: Skipped 1 previous similar message [ 6194.662956] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6198.398495] Lustre: server umount lustre-MDT0001 complete [ 6199.371120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6199.388682] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.203.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.392386] LustreError: Skipped 1 previous similar message [ 6201.911631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.945802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6201.948592] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6201.963737] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6202.083394] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6203.582160] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 6207.459951] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6207.462705] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6211.556757] LustreError: 310757:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 6211.559386] LustreError: 310757:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6211.561531] LustreError: 310757:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6211.566544] LustreError: 281838:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6211.569373] LustreError: 281838:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 6211.586465] LustreError: 310760:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 6211.592607] LustreError: 310760:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.203.142@tcp: cfg command failed: rc = -114 [ 6211.595711] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 6211.595711] [ 6519.777254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.778337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6519.781238] Lustre: Skipped 3 previous similar messages [ 6519.783769] Lustre: Skipped 3 previous similar messages [ 6519.785060] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6524.103199] LustreError: 322327:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000f97d0a85) refcount nonzero (0) after lock cleanup; forcing cleanup. [ 6524.113880] Lustre: server umount lustre-MDT0000 complete [ 6524.896338] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.900765] LustreError: Skipped 4 previous similar messages [ 6525.402159] LustreError: 285104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764943535 with bad export cookie 2303899741377837457 [ 6525.405376] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6525.406326] LustreError: 285104:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6530.016209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.023607] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6531.569192] Lustre: server umount lustre-MDT0001 complete [ 6535.994764] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6538.928584] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6542.429790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6544.718117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6546.766274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6548.911619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.083821] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 09:06:01 (1764943561) [ 6553.244985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6553.271600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.350063] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6553.360266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6553.399512] Lustre: lustre-MDT0000: new disk, initializing [ 6553.428233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.435768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6554.856748] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 6558.969944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6558.997616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.028775] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6559.042825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6559.045659] Lustre: Skipped 1 previous similar message [ 6559.078825] Lustre: lustre-MDT0001: new disk, initializing [ 6559.107078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6559.120792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6559.123840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6560.499517] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 6564.382741] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6566.369170] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6568.575945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6568.596708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.600353] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6568.690334] Lustre: lustre-OST0000: new disk, initializing [ 6568.693073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6568.719557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6569.851947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6569.856039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6570.143824] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_default_debug -1 all 8 [ 6574.039563] Lustre: DEBUG MARKER: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6579.679660] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6579.681969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.686945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6579.689158] Lustre: Skipped 1 previous similar message [ 6581.797866] Lustre: server umount lustre-MDT0000 complete [ 6581.799871] Lustre: Skipped 1 previous similar message [ 6583.067311] LustreError: 325368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764943593 with bad export cookie 2303899741377839354 [ 6583.069024] LustreError: 326238:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 6583.069780] LustreError: 166-1: MGC192.168.203.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6583.072513] LustreError: 325368:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6583.084783] LustreError: 326238:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 6583.089627] LustreError: 326238:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 6589.919137] Lustre: 328626:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764943594/real 1764943594] req@000000001c402ab7 x1850676391094528/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764943600 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6598.731082] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing set_hostid [ 6601.536816] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing load_modules_local [ 6605.032495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6607.713164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6609.659775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6611.755153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6612.261669] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6396 sec ========== 09:07:02 (1764943622) [ 6625.951483] Lustre: DEBUG MARKER: oleg342-server.virtnet: executing unload_modules_local [ 6626.913906] Key type lgssc unregistered [ 6627.020334] LNet: 333719:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6627.023266] LNet: Removed LNI 192.168.203.142@tcp [ 6627.333743] Key type .llcrypt unregistered [ 6627.334929] Key type ._llcrypt unregistered