[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487015719 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002387] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007439] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014091] *** VALIDATE selinux *** [ 0.022584] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025035] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028126] *** VALIDATE tmpfs *** [ 0.030190] *** VALIDATE proc *** [ 0.031240] *** VALIDATE cgroup *** [ 0.032017] *** VALIDATE cgroup2 *** [ 0.034025] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036109] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039016] Speculative Store Bypass: Vulnerable [ 0.042375] debug: unmapping init [mem 0xffffffffb0e59000-0xffffffffb0e60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045708] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053302] rcu: Hierarchical SRCU implementation. [ 0.055453] smp: Bringing up secondary CPUs ... [ 0.056582] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.061206] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139903] node 0 deferred pages initialised in 71ms [ 0.143108] devtmpfs: initialized [ 0.144205] x86/mm: Memory block size: 128MB [ 0.146632] gcov: version magic: 0x41383552 [ 0.149295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154264] pinctrl core: initialized pinctrl subsystem [ 0.155171] [ 0.155619] ************************************************************* [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160008] ** ** [ 0.161009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163010] ** ** [ 0.165010] ** This means that this kernel is built to expose internal ** [ 0.167010] ** IOMMU data structures, which may compromise security on ** [ 0.169010] ** your system. ** [ 0.170008] ** ** [ 0.172010] ** If you see this message and you are not debugging the ** [ 0.174010] ** kernel, report this immediately to your vendor! ** [ 0.176010] ** ** [ 0.178011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179008] ************************************************************* [ 0.181608] NET: Registered protocol family 16 [ 0.183415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.185054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.187052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190108] cpuidle: using governor menu [ 0.191956] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194455] PCI: Using configuration type 1 for base access [ 0.196146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208060] cryptd: max_cpu_qlen set to 1000 [ 0.210247] ACPI: Added _OSI(Module Device) [ 0.212015] ACPI: Added _OSI(Processor Device) [ 0.213010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219396] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225524] ACPI: Interpreter enabled [ 0.226065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228012] ACPI: Using IOAPIC for interrupt routing [ 0.229095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242203] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255205] acpiphp: Slot [2] registered [ 0.256125] acpiphp: Slot [3] registered [ 0.258084] acpiphp: Slot [4] registered [ 0.259073] acpiphp: Slot [5] registered [ 0.260073] acpiphp: Slot [6] registered [ 0.261005] acpiphp: Slot [7] registered [ 0.261946] acpiphp: Slot [8] registered [ 0.262074] acpiphp: Slot [9] registered [ 0.262971] acpiphp: Slot [10] registered [ 0.265113] acpiphp: Slot [11] registered [ 0.266117] acpiphp: Slot [12] registered [ 0.268088] acpiphp: Slot [13] registered [ 0.269090] acpiphp: Slot [14] registered [ 0.270091] acpiphp: Slot [15] registered [ 0.272106] acpiphp: Slot [16] registered [ 0.273100] acpiphp: Slot [17] registered [ 0.275096] acpiphp: Slot [18] registered [ 0.276121] acpiphp: Slot [19] registered [ 0.278104] acpiphp: Slot [20] registered [ 0.279000] acpiphp: Slot [21] registered [ 0.279000] acpiphp: Slot [22] registered [ 0.280093] acpiphp: Slot [23] registered [ 0.282118] acpiphp: Slot [24] registered [ 0.283111] acpiphp: Slot [25] registered [ 0.285106] acpiphp: Slot [26] registered [ 0.286070] acpiphp: Slot [27] registered [ 0.287079] acpiphp: Slot [28] registered [ 0.289115] acpiphp: Slot [29] registered [ 0.290070] acpiphp: Slot [30] registered [ 0.291099] acpiphp: Slot [31] registered [ 0.292061] PCI host bridge to bus 0000:00 [ 0.293014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.302021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311253] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320941] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333431] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.335677] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.338036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.340553] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343865] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.354009] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.359016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364195] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.379018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.395026] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.404166] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.410016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.429026] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.438836] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.446016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.451015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.464020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.472998] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.482016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.487018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.497016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.506282] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.513013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.519015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.541016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.554049] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.559019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.562016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.574027] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.585491] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.587392] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.589372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.592370] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.594233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.599561] iommu: Default domain type: Passthrough [ 0.600308] SCSI subsystem initialized [ 0.601000] ACPI: bus type USB registered [ 0.603132] usbcore: registered new interface driver usbfs [ 0.605090] usbcore: registered new interface driver hub [ 0.607080] usbcore: registered new device driver usb [ 0.609176] pps_core: LinuxPPS API ver. 1 registered [ 0.610010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.614059] PTP clock support registered [ 0.616132] EDAC MC: Ver: 3.0.0 [ 0.617117] PCI: Using ACPI for IRQ routing [ 0.618000] NetLabel: Initializing [ 0.618010] NetLabel: domain hash size = 128 [ 0.619009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.620077] NetLabel: unlabeled traffic allowed by default [ 0.623093] vgaarb: loaded [ 0.624203] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.625014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.630319] clocksource: Switched to clocksource kvm-clock [ 0.734075] VFS: Disk quotas dquot_6.6.0 [ 0.735828] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.738418] *** VALIDATE ramfs *** [ 0.739652] *** VALIDATE hugetlbfs *** [ 0.741276] pnp: PnP ACPI init [ 0.743423] pnp: PnP ACPI: found 6 devices [ 0.765071] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.769543] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772431] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.776120] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.780025] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.783137] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.786810] NET: Registered protocol family 2 [ 0.789550] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.795452] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.800182] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.806542] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.810774] TCP: Hash tables configured (established 65536 bind 65536) [ 0.814348] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.817816] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.821524] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.825431] NET: Registered protocol family 1 [ 0.828332] RPC: Registered named UNIX socket transport module. [ 0.830465] RPC: Registered udp transport module. [ 0.831841] RPC: Registered tcp transport module. [ 0.833229] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.835718] NET: Registered protocol family 44 [ 0.837323] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.839204] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.841260] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.843617] PCI: CLS 0 bytes, default 64 [ 0.845681] Unpacking initramfs... [ 2.259455] debug: unmapping init [mem 0xffff92ff7cc54000-0xffff92ff7ffbffff] [ 2.263627] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.265887] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.268898] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.789218] Initialise system trusted keyrings [ 2.791062] Key type blacklist registered [ 2.793244] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.803756] zbud: loaded [ 2.807562] *** VALIDATE nfs *** [ 2.809425] *** VALIDATE nfs4 *** [ 2.811425] pstore: using deflate compression [ 2.814383] Platform Keyring initialized [ 2.945255] NET: Registered protocol family 38 [ 2.947347] Key type asymmetric registered [ 2.948743] Asymmetric key parser 'x509' registered [ 2.950885] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.955596] io scheduler mq-deadline registered [ 2.957652] io scheduler kyber registered [ 2.964799] io scheduler bfq registered [ 2.966804] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.970022] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.973341] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.975538] ACPI: Power Button [PWRF] [ 3.070175] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.165845] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.345281] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.436456] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.620866] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.654118] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.682917] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.686750] Non-volatile memory driver v1.3 [ 3.688114] Linux agpgart interface v0.103 [ 3.715799] virtio_blk virtio1: [vda] 67984 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.718394] vda: detected capacity change from 0 to 34807808 [ 3.732485] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.735467] vdb: detected capacity change from 0 to 1073741824 [ 3.749275] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.752019] vdc: detected capacity change from 0 to 2621440000 [ 3.767515] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.770326] vdd: detected capacity change from 0 to 2621440000 [ 3.786231] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.788792] vde: detected capacity change from 0 to 4294967296 [ 3.805844] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.808367] vdf: detected capacity change from 0 to 4294967296 [ 3.814062] libphy: Fixed MDIO Bus: probed [ 3.820566] usbcore: registered new interface driver usbserial_generic [ 3.823120] usbserial: USB Serial support registered for generic [ 3.825380] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.829660] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.831330] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.833857] mousedev: PS/2 mouse device common for all mice [ 3.836362] rtc_cmos 00:05: RTC can wake from S4 [ 3.839020] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.841691] rtc_cmos 00:05: registered as rtc0 [ 3.844236] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.846759] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.848322] intel_pstate: CPU model not supported [ 3.855120] hid: raw HID events driver (C) Jiri Kosina [ 3.856950] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.857379] usbcore: registered new interface driver usbhid [ 3.861335] usbhid: USB HID core driver [ 3.862853] drop_monitor: Initializing network drop monitor service [ 3.865868] Initializing XFRM netlink socket [ 3.867851] NET: Registered protocol family 10 [ 3.871533] Segment Routing with IPv6 [ 3.873213] NET: Registered protocol family 17 [ 3.875270] mpls_gso: MPLS GSO support [ 3.880777] RAS: Correctable Errors collector initialized. [ 3.882724] AVX version of gcm_enc/dec engaged. [ 3.884189] AES CTR mode by8 optimization enabled [ 3.960484] sched_clock: Marking stable (3960461789, 0)->(4876779593, -916317804) [ 3.963953] registered taskstats version 1 [ 3.966272] Loading compiled-in X.509 certificates [ 3.968469] zswap: loaded using pool lzo/zbud [ 3.994388] Key type big_key registered [ 4.011792] Key type encrypted registered [ 4.013583] ima: No TPM chip found, activating TPM-bypass! [ 4.015397] ima: Allocated hash algorithm: sha1 [ 4.016775] ima: No architecture policies found [ 4.018161] evm: Initialising EVM extended attributes: [ 4.019646] evm: security.selinux [ 4.020603] evm: security.ima [ 4.021484] evm: security.capability [ 4.022464] evm: HMAC attrs: 0x1 [ 4.024643] rtc_cmos 00:05: setting system clock to 2025-10-24 13:39:48 UTC (1761313188) [ 4.031190] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 4.033798] debug: unmapping init [mem 0xffffffffb0b82000-0xffffffffb0e58fff] [ 4.043108] Write protecting the kernel read-only data: 28672k [ 4.046046] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 4.048151] debug: unmapping init [mem 0xffffffffafb14000-0xffffffffafbfffff] [ 4.084184] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.090563] systemd[1]: Detected virtualization kvm. [ 4.092506] systemd[1]: Detected architecture x86-64. [ 4.094165] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.118601] systemd[1]: No hostname configured. [ 4.121604] systemd[1]: Set hostname to . [ 4.123857] random: systemd: uninitialized urandom read (16 bytes read) [ 4.126297] systemd[1]: Initializing machine ID from random generator. [ 4.166729] random: ln: uninitialized urandom read (6 bytes read) [ 4.251831] random: systemd: uninitialized urandom read (16 bytes read) [ 4.254048] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.257587] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.261963] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.841519] device-mapper: uevent: version 1.0.3 [ 4.843606] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.569491] virtio_net virtio0 ens2: renamed from eth0 [ 5.603511] random: fast init done [ 5.625406] scsi host0: ata_piix [ 5.667594] scsi host1: ata_piix [ 5.669458] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.671789] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.389237] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.527049] random: crng init done [ 10.528459] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.978672] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.222601] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.808800] SELinux: Disabled at runtime. [ 12.918795] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.937946] systemd[1]: Detected virtualization kvm. [ 12.939754] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.183809] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.188255] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.199219] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.203515] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.205903] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.220527] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.239334] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ 14.832087] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 15.506530] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.393360] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.412893] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.249161] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.294706] EDAC sbridge: Ver: 1.1.2 [ 20.414314] Key type dns_resolver registered [ 21.195972] NFS: Registering the id_resolver key type [ 21.202879] Key type id_resolver registered [ 21.204557] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 56.637797] libcfs: loading out-of-tree module taints kernel. [ 56.668143] alg: No test for adler32 (adler32-zlib) [ 57.421423] Key type ._llcrypt registered [ 57.423070] Key type .llcrypt registered [ 57.486192] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 69.254531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 70.461425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.933614] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 71.546465] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 71.550283] LNet: Accept secure, port 988 [ 73.223930] Key type lgssc registered [ 74.373292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.069033] hrtimer: interrupt took 4159813 ns [ 87.367113] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 89.149659] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.677843] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.974423] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.297186] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.899525] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 133.852633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.953273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.203329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 135.222957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 135.269652] Lustre: lustre-MDT0000: new disk, initializing [ 135.339611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.357423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.301921] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 148.798268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.854596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.896369] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 148.914557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 148.918175] Lustre: Skipped 1 previous similar message [ 149.001972] Lustre: lustre-MDT0001: new disk, initializing [ 149.064713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 149.090519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 149.105338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 152.783095] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 164.733629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.775718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 164.781950] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 164.917079] Lustre: lustre-OST0000: new disk, initializing [ 164.919281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 164.953473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.969991] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 175.120985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.134662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 178.480664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.544292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.549698] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 178.603728] Lustre: lustre-OST0001: new disk, initializing [ 178.606821] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 178.659672] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.397336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 181.409070] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 182.218313] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 192.904253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 204.105712] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 212.672562] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 217.486568] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 221.362512] Lustre: DEBUG MARKER: Client: 2.15.7.13 [ 223.449574] Lustre: DEBUG MARKER: MDS: 2.15.7.13 [ 225.483756] Lustre: DEBUG MARKER: OSS: 2.15.7.13 [ 227.216820] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:43:29 EDT 2025 [ 235.173378] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball 110 [ 236.886188] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 251.359830] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 251.371656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.380145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 253.409304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 253.413051] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.418750] Lustre: Skipped 1 previous similar message [ 253.429706] Lustre: Skipped 1 previous similar message [ 257.125511] Lustre: server umount lustre-MDT0000 complete [ 258.527713] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.539271] LustreError: Skipped 3 previous similar messages [ 259.790078] LustreError: 6219:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313444 with bad export cookie 7917352884458289254 [ 259.792607] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.799041] LustreError: 6219:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 263.650417] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 263.653803] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 263.655245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 263.655252] Lustre: Skipped 2 previous similar messages [ 263.661578] Lustre: Skipped 2 previous similar messages [ 263.682942] LustreError: Skipped 1 previous similar message [ 264.671627] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.690653] LustreError: Skipped 1 previous similar message [ 265.701251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 265.703946] Lustre: Skipped 3 previous similar messages [ 270.816178] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.816194] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 270.820080] LustreError: Skipped 2 previous similar messages [ 270.828400] Lustre: Skipped 1 previous similar message [ 274.399220] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 274.614986] Lustre: server umount lustre-MDT0001 complete [ 277.795921] Lustre: server umount lustre-OST0000 complete [ 281.250118] Lustre: server umount lustre-OST0001 complete [ 287.025369] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 294.424774] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 302.774946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.290111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.972879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 317.523337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.133896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.209556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.408392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 323.433405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 323.484823] Lustre: lustre-MDT0000: new disk, initializing [ 323.552489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.569977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 327.338115] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 335.752438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.817151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 335.915903] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 335.968337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 335.971560] Lustre: Skipped 1 previous similar message [ 336.026413] Lustre: lustre-MDT0001: new disk, initializing [ 336.110508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 336.136761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 336.147961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 339.572675] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 348.134635] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 353.089762] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 358.844682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.905354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 358.911842] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 359.131362] Lustre: lustre-OST0000: new disk, initializing [ 359.134285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 359.204885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.261354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 360.268167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 362.724530] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 371.859339] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 375.749149] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 375.925537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 379.693043] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 379.858393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 386.017761] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 386.019028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.021261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 386.028427] LustreError: Skipped 1 previous similar message [ 386.043809] Lustre: Skipped 1 previous similar message [ 387.407479] Lustre: server umount lustre-OST0000 complete [ 391.136946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.383342] Lustre: server umount lustre-MDT0000 complete [ 398.989261] LustreError: 14307:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313583 with bad export cookie 7917352884458290717 [ 398.999187] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.017495] LustreError: 3335:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 403.133858] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:46:26 (1761313586) [ 407.358424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.641503] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.648785] LustreError: Skipped 1 previous similar message [ 407.701239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.511432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 417.091629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.830683] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 428.591600] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 433.473792] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 438.354331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.602584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 446.276204] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 465.153852] Lustre: Modifying parameter general.osc.*.max_dirty_mb in log params [ 473.569408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.569511] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 473.574871] Lustre: Skipped 1 previous similar message [ 473.578974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 473.591341] Lustre: Skipped 7 previous similar messages [ 475.357331] Lustre: server umount lustre-MDT0000 complete [ 475.362315] Lustre: Skipped 1 previous similar message [ 477.763852] LustreError: 20668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313662 with bad export cookie 7917352884458291564 [ 477.766416] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.778481] LustreError: 20668:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 485.859240] Lustre: 21845:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313664/real 1761313664] req@000000004e3563bb x1846870808803392/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313670 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 501.247319] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 508.641475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.912942] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.922191] LustreError: Skipped 1 previous similar message [ 508.954208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.960727] Lustre: Skipped 2 previous similar messages [ 512.021065] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 519.712519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.559841] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 531.285798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 535.202641] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 541.162227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 541.202468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.207130] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 541.302672] Lustre: lustre-OST0001: new disk, initializing [ 541.305388] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 542.705048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 542.709155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 544.361974] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 552.860747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 555.955566] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 568.290649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.298974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 568.301669] Lustre: Skipped 4 previous similar messages [ 568.304429] Lustre: Skipped 1 previous similar message [ 574.267797] Lustre: server umount lustre-MDT0000 complete [ 574.287135] Lustre: Skipped 2 previous similar messages [ 576.758444] LustreError: 24798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313761 with bad export cookie 7917352884458292418 [ 576.763536] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.764652] LustreError: 24798:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 585.631152] Lustre: 3335:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313763/real 1761313763] req@000000001e024aa4 x1846870808832064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313770 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 588.769667] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313766/real 1761313766] req@000000008efe7197 x1846870808832256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313773 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 588.799845] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 601.811618] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 608.313453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.590339] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.596038] LustreError: Skipped 2 previous similar messages [ 611.682695] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 617.703136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.104875] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 626.969751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 629.828331] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 635.433742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 638.519973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 644.188724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 646.573702] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 646.576485] Lustre: Skipped 1 previous similar message [ 658.911915] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 658.918900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.931145] Lustre: Skipped 2 previous similar messages [ 658.939905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.942738] Lustre: Skipped 6 previous similar messages [ 661.879700] Lustre: server umount lustre-MDT0000 complete [ 661.882163] Lustre: Skipped 3 previous similar messages [ 664.419901] LustreError: 28450:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313848 with bad export cookie 7917352884458293657 [ 664.420931] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 664.431491] LustreError: 28450:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 672.159308] Lustre: 3334:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313849/real 1761313849] req@0000000055965774 x1846870808854272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313856 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 682.535689] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:51:05 (1761313865) [ 703.098286] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 711.016171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.273362] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.282996] LustreError: Skipped 2 previous similar messages [ 711.345174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.350064] Lustre: Skipped 7 previous similar messages [ 714.172424] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 719.886489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.212618] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 728.893958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 732.268122] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 738.307615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.405310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 747.417425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 750.496601] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 761.314631] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.325978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.342586] Lustre: Skipped 4 previous similar messages [ 768.571852] LustreError: 35114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761313953 with bad export cookie 7917352884458294735 [ 768.577421] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.582090] LustreError: 35114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 778.207200] Lustre: 37859:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313956/real 1761313956] req@000000005748c4e8 x1846870808876800/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761313962 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 778.250884] Lustre: 37859:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 790.256815] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:52:52 (1761313972) [ 802.909077] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 810.749823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.170841] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.180206] LustreError: Skipped 6 previous similar messages [ 814.823073] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 821.875670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.889864] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 831.631791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 835.148409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 841.403642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 844.288312] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 850.613287] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 861.727218] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 861.730181] Lustre: Skipped 9 previous similar messages [ 863.202426] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 867.110684] Lustre: server umount lustre-MDT0000 complete [ 867.113261] Lustre: Skipped 7 previous similar messages [ 870.314491] LustreError: 39467:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314054 with bad export cookie 7917352884458295806 [ 870.319438] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.328249] LustreError: 39467:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 882.132942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.752944] Lustre: Evicted from MGS (at 192.168.204.154@tcp) after server handle changed from 0x6de016801f8d11fe to 0x6de016801f8d1665 [ 886.764621] Lustre: MGC192.168.204.154@tcp: Connection restored to (at 0@lo) [ 887.100979] BUG: spinlock bad magic on CPU#2, lctl/43398 [ 887.102597] lock: 0xffff92fffccae900, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 887.106376] CPU: 2 PID: 43398 Comm: lctl Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 887.110991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 887.117428] Call Trace: [ 887.118435] ? dump_stack+0xbb/0x10e [ 887.120172] ? spin_dump.cold.3+0x32/0x65 [ 887.121930] ? do_raw_spin_lock+0xce/0x190 [ 887.125913] ? _raw_spin_lock+0x21/0x30 [ 887.129846] ? mdd_changelog_mask_seq_write+0x61a/0x660 [mdd] [ 887.132752] ? __seccomp_filter+0x4b/0x680 [ 887.134243] ? full_proxy_write+0x5e/0xa0 [ 887.135871] ? __vfs_write+0x1c/0x60 [ 887.137177] ? vfs_write+0xd8/0x2b0 [ 887.138456] ? ksys_write+0x66/0x120 [ 887.139686] ? __x64_sys_write+0x1e/0x30 [ 887.141129] ? do_syscall_64+0xc1/0x440 [ 887.142663] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 889.990943] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 895.684484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.307410] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 903.149579] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 905.055890] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 906.446462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 907.244818] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 907.247913] Lustre: Skipped 1 previous similar message [ 910.499860] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 911.951517] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 916.452711] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 916.458051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.464242] Lustre: Skipped 10 previous similar messages [ 924.437431] LustreError: 39466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314108 with bad export cookie 7917352884458296933 [ 924.441502] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.448732] LustreError: 39466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 933.343142] Lustre: 45615:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314111/real 1761314111] req@000000004954ae27 x1846870808916160/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314117 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 933.358164] Lustre: 45615:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 943.801418] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:55:26 (1761314126) [ 955.881076] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 962.669506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.926077] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.934657] LustreError: Skipped 24 previous similar messages [ 966.056272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 972.453602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.814244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 972.818455] Lustre: Skipped 10 previous similar messages [ 975.802282] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 981.879869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.110682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 992.030379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.354078] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1001.696289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1004.782903] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 1004.787500] Lustre: Skipped 2 previous similar messages [ 1018.850320] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.131980] LustreError: 47224:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314211 with bad export cookie 7917352884458297745 [ 1027.135344] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.143718] LustreError: 47224:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1036.255153] Lustre: 50815:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314214/real 1761314214] req@00000000f4b3eca2 x1846870808941632/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761314220 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1036.278919] Lustre: 50815:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1045.888380] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:57:08 (1761314228) [ 1047.357541] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1048.504636] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:57:11 (1761314231) [ 1056.072958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1061.552302] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1068.165474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1073.456544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1078.277682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1078.337290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.519798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1078.568657] Lustre: lustre-MDT0000: new disk, initializing [ 1078.677267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1081.831650] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1089.328042] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1093.293164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1093.330195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1093.334381] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1095.556822] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1101.538835] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1102.897450] Lustre: DEBUG MARKER: create test files [ 1125.346587] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1125.349748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.352303] Lustre: Skipped 20 previous similar messages [ 1127.511596] Lustre: server umount lustre-OST0000 complete [ 1127.514012] Lustre: Skipped 9 previous similar messages [ 1136.607228] Lustre: 55526:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314315/real 1761314315] req@00000000e390af95 x1846870808950208/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314321 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 1136.627116] Lustre: 55526:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1144.312066] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1145.983541] Key type lgssc unregistered [ 1146.138514] LNet: 56129:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.143781] LNet: Removed LNI 192.168.204.154@tcp [ 1146.665208] Key type .llcrypt unregistered [ 1146.667141] Key type ._llcrypt unregistered [ 1164.378468] alg: No test for adler32 (adler32-zlib) [ 1165.133703] Key type ._llcrypt registered [ 1165.136092] Key type .llcrypt registered [ 1165.223892] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1166.015445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1166.208696] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1166.339791] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1166.343166] LNet: Accept secure, port 988 [ 1167.959396] Key type lgssc registered [ 1168.692990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1174.203637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.498593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.407630] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1184.269694] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1189.249632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1189.482070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.529479] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 1191.584124] Lustre: 58972:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1191.588661] Lustre: 58972:0:(osd_handler.c:1953:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1191.592023] Lustre: 58972:0:(osd_handler.c:1960:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1191.595664] Lustre: 58972:0:(osd_handler.c:1970:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1191.600104] Lustre: 58972:0:(osd_handler.c:1977:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1191.604771] Lustre: 58972:0:(osd_handler.c:1984:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1191.844570] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1196.017838] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1199.860469] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1203.593387] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1212.388219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.389708] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.414787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.650504] Lustre: server umount lustre-OST0000 complete [ 1221.600187] Lustre: 59984:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314400/real 1761314400] req@000000000b6d1af9 x1846871955871552/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314406 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1221.692479] Lustre: server umount lustre-MDT0000 complete [ 1229.101709] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1230.821834] Key type lgssc unregistered [ 1231.056191] LNet: 60587:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.063313] LNet: Removed LNI 192.168.204.154@tcp [ 1231.562380] Key type .llcrypt unregistered [ 1231.564084] Key type ._llcrypt unregistered [ 1251.211515] alg: No test for adler32 (adler32-zlib) [ 1251.963718] Key type ._llcrypt registered [ 1251.966511] Key type .llcrypt registered [ 1252.052283] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1252.785389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1252.993506] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1253.139104] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1253.142044] LNet: Accept secure, port 988 [ 1254.759140] Key type lgssc registered [ 1255.412816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.750929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.106676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.325904] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1270.603567] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1274.769123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1275.034489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.103667] Lustre: lustre-OST0000: deleting orphan objects from 0x0:140 to 0x0:161 [ 1277.177850] Lustre: 63524:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1277.183353] Lustre: 63524:0:(osd_handler.c:1953:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1277.186685] Lustre: 63524:0:(osd_handler.c:1960:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1277.190216] Lustre: 63524:0:(osd_handler.c:1970:osd_trans_dump_creds()) write: 1/1/1, punch: 0/0/0, quota 1/3/0 [ 1277.194055] Lustre: 63524:0:(osd_handler.c:1977:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1277.198654] Lustre: 63524:0:(osd_handler.c:1984:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1278.457117] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1284.350983] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1288.933705] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1297.889690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1297.892371] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1297.899117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1300.632868] Lustre: server umount lustre-OST0000 complete [ 1308.639174] Lustre: 64499:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314487/real 1761314487] req@00000000195fb69b x1846872047097856/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314493 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 1308.775346] Lustre: server umount lustre-MDT0000 complete [ 1316.470310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1318.072811] Key type lgssc unregistered [ 1318.244724] LNet: 65102:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.250500] LNet: Removed LNI 192.168.204.154@tcp [ 1318.680924] Key type .llcrypt unregistered [ 1318.683597] Key type ._llcrypt unregistered [ 1331.193847] alg: No test for adler32 (adler32-zlib) [ 1331.947612] Key type ._llcrypt registered [ 1331.948939] Key type .llcrypt registered [ 1332.045321] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1342.353883] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1343.066554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.246971] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 1343.390640] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1343.397155] LNet: Accept secure, port 988 [ 1345.032121] Key type lgssc registered [ 1345.740995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.827484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1356.666918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1361.350604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1367.098552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1373.641839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1373.716803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.961176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1374.983531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1375.048512] Lustre: lustre-MDT0000: new disk, initializing [ 1375.154468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.180531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1379.824112] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1388.704614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1388.792982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.884814] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1388.975199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1388.985337] Lustre: Skipped 1 previous similar message [ 1389.055509] Lustre: lustre-MDT0001: new disk, initializing [ 1389.150200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.182778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1389.194269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1393.499636] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1401.454598] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1405.993175] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1411.254505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1411.338716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.345665] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1411.559728] Lustre: lustre-OST0000: new disk, initializing [ 1411.577424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1411.651170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.922320] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1420.838552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1420.843584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1422.794328] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1427.190943] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1427.386444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1430.706804] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1430.950961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1436.128890] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.136675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.151119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.155679] Lustre: Skipped 1 previous similar message [ 1438.846642] Lustre: server umount lustre-OST0000 complete [ 1445.349738] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1445.354783] LustreError: Skipped 1 previous similar message [ 1445.358799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.364189] Lustre: Skipped 1 previous similar message [ 1445.367543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.368657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.369229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.537795] Lustre: server umount lustre-MDT0000 complete [ 1451.003355] LustreError: 68374:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314635 with bad export cookie 8164434960316165239 [ 1451.004625] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.012568] LustreError: 68374:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1451.311831] Lustre: server umount lustre-MDT0001 complete [ 1457.187879] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:04:00 (1761314640) [ 1464.630679] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1473.686169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1492.694069] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1497.916667] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1503.991810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1508.091197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1512.737348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1517.923652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.939298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1524.035815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.272256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1524.297172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1524.353457] Lustre: lustre-MDT0000: new disk, initializing [ 1524.422673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.435594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1528.522348] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1537.124518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1537.199944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.281515] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1537.323474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 1537.326585] Lustre: Skipped 1 previous similar message [ 1537.386858] Lustre: lustre-MDT0001: new disk, initializing [ 1537.500353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1537.537475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1537.551841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1540.954638] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1549.946943] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1554.912211] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1561.366058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1561.449369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1561.458326] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1561.627178] Lustre: lustre-OST0000: new disk, initializing [ 1561.629973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 1561.708713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1565.481810] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1568.809908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1568.821633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1573.866983] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1577.626777] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1577.788786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1581.246804] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1581.429593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1584.101407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.102169] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1584.103614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1584.103621] Lustre: Skipped 1 previous similar message [ 1584.109469] Lustre: Skipped 1 previous similar message [ 1589.041916] Lustre: server umount lustre-OST0000 complete [ 1593.830099] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1593.833651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1593.840847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.847559] Lustre: Skipped 1 previous similar message [ 1598.260800] Lustre: server umount lustre-MDT0000 complete [ 1599.460323] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.482870] LustreError: Skipped 1 previous similar message [ 1600.487278] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.500021] LustreError: Skipped 1 previous similar message [ 1601.244348] LustreError: 76662:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314785 with bad export cookie 8164434960316166093 [ 1601.251074] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.252962] LustreError: 76662:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1608.125936] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:06:31 (1761314791) [ 1613.466660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.858468] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.917177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.105713] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1618.915155] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.659722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.228244] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1634.397610] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1640.427631] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1647.064596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1647.301813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.310574] Lustre: Skipped 1 previous similar message [ 1651.051841] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1657.808588] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1660.920031] Lustre: 83960:0:(genops.c:1710:obd_export_evict_by_uuid()) MGS: evicting 5673eb03-2810-4fe4-bd2c-1e66b343699c at adminstrative request [ 1663.718160] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1663.733802] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1670.119139] Lustre: MGS: Client 5673eb03-2810-4fe4-bd2c-1e66b343699c (at 0@lo) reconnecting [ 1670.127252] Lustre: Evicted from MGS (at 192.168.204.154@tcp) after server handle changed from 0x714de656e16cf323 to 0x714de656e16cf5c3 [ 1670.139617] Lustre: MGC192.168.204.154@tcp: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 1695.894884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1695.963595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1695.966371] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 1696.040797] Lustre: lustre-OST0001: new disk, initializing [ 1696.043740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 1698.951440] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1702.450951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1702.458218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1705.907680] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1708.000094] LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.001469] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1708.005873] LustreError: Skipped 1 previous similar message [ 1708.014448] Lustre: Skipped 1 previous similar message [ 1708.016401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1708.016407] Lustre: Skipped 2 previous similar messages [ 1722.335586] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1722.423768] Lustre: server umount lustre-OST0001 complete [ 1722.425509] Lustre: Skipped 1 previous similar message [ 1722.851176] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.859532] LustreError: Skipped 1 previous similar message [ 1726.435715] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1726.444949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.466937] Lustre: Skipped 1 previous similar message [ 1726.483339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1726.488760] Lustre: Skipped 5 previous similar messages [ 1733.091538] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.107690] LustreError: Skipped 3 previous similar messages [ 1740.257560] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1740.447049] Lustre: server umount lustre-OST0000 complete [ 1746.912459] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1753.568279] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1753.583116] LustreError: Skipped 3 previous similar messages [ 1754.542051] LustreError: 81202:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761314939 with bad export cookie 8164434960316167619 [ 1754.560540] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.206963] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:09:06 (1761314946) [ 1764.469968] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1765.835483] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:09:08 (1761314948) [ 1767.123779] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1768.575662] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:09:11 (1761314951) [ 1769.735896] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1771.128648] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:09:14 (1761314954) [ 1772.360707] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1775.975879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1776.746839] loop: module loaded [ 1776.757223] loop0: detected capacity change from 0 to 4294967296 [ 1776.798876] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1777.954091] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1780.171917] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1782.568666] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1788.250694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1791.469931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.532447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1798.822658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1805.581427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1805.723737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.734277] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1805.781245] LustreError: 88412:0:(obd_config.c:774:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1805.790721] LustreError: 88412:0:(obd_mount.c:200:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1805.811207] LustreError: 88412:0:(obd_mount_server.c:1993:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1805.820941] LustreError: 88412:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1823.513353] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1831.875179] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1840.079027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1845.502209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1850.645151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1856.615773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1862.324007] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1862.377368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.508575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1862.530612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 1862.604042] Lustre: lustre-MDT0000: new disk, initializing [ 1862.675671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.679551] Lustre: Skipped 1 previous similar message [ 1862.696752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1866.981329] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1877.354839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1877.398568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.468811] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 1877.629303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1881.856408] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1889.623644] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1894.812741] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1901.294960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1901.356514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1901.360371] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 1905.487232] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1909.828870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1913.786782] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1917.638549] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 1917.827196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1921.542700] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 1921.735742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1925.603960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.604379] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1925.616422] Lustre: Skipped 3 previous similar messages [ 1925.624774] LustreError: Skipped 1 previous similar message [ 1925.634047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1925.641538] Lustre: Skipped 9 previous similar messages [ 1929.489858] Lustre: server umount lustre-OST0000 complete [ 1929.491754] Lustre: Skipped 2 previous similar messages [ 1940.960378] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.966034] LustreError: Skipped 1 previous similar message [ 1941.210577] LustreError: 91267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315125 with bad export cookie 8164434960316168193 [ 1941.212648] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.220351] LustreError: 91267:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1947.501194] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:12:10 (1761315130) [ 1953.668711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.223615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1966.724974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.806941] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 1980.150866] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1985.478660] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1990.908026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1991.192122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.195681] Lustre: Skipped 4 previous similar messages [ 1994.809239] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2000.510836] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2006.329479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2006.398388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.404682] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2006.525687] Lustre: lustre-OST0001: new disk, initializing [ 2006.527988] Lustre: Skipped 2 previous similar messages [ 2006.536538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2006.544728] Lustre: Skipped 3 previous similar messages [ 2009.958196] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2015.297244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2015.302427] Lustre: Skipped 2 previous similar messages [ 2015.306479] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2017.644943] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2034.430924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2054.500159] Lustre: Failing over lustre-MDT0000 [ 2054.781526] Lustre: server umount lustre-MDT0000 complete [ 2054.783906] Lustre: Skipped 2 previous similar messages [ 2055.733566] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.744908] LustreError: Skipped 4 previous similar messages [ 2056.673508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.687189] Lustre: Skipped 5 previous similar messages [ 2063.839181] Lustre: 66599:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315241/real 1761315241] req@000000007c3228a8 x1846872141585088/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315248 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2063.854303] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.290673] LDISKFS-fs (dm-0): recovery complete [ 2076.302505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.254689] Lustre: Evicted from MGS (at 192.168.204.154@tcp) after server handle changed from 0x714de656e16cfb5e to 0x714de656e16d8948 [ 2081.264699] Lustre: MGC192.168.204.154@tcp: Connection restored to (at 0@lo) [ 2081.569490] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2084.369310] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2084.394947] Lustre: 95821:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2085.204966] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2086.880854] Lustre: 95826:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2086.905219] Lustre: 95826:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 1 previous similar message [ 2086.916612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2087.001518] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2087.319641] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2087.670273] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2087.686433] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2088.007163] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2088.010720] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 2088.017727] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2088.027545] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 7 previous similar messages [ 2088.977807] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2088.986221] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 2089.319343] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2089.322905] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 2090.281484] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2090.296536] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 6 previous similar messages [ 2091.256075] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2091.264810] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2091.602289] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2091.612067] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 2094.508899] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2094.526319] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 12 previous similar messages [ 2095.499745] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2095.509824] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2095.839141] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2095.851281] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages [ 2102.535593] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2102.551731] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 24 previous similar messages [ 2103.512277] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2103.524509] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2103.848549] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2103.861151] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 24 previous similar messages [ 2118.802809] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2118.820597] Lustre: 101004:0:(ldlm_lib.c:1970:extend_recovery_timer()) Skipped 50 previous similar messages [ 2119.784625] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms [ 2119.791788] LustreError: 101004:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2120.111164] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake [ 2120.115314] LustreError: 101004:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 50 previous similar messages [ 2144.248090] Lustre: 101004:0:(ldlm_lib.c:2278:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2144.254481] Lustre: 101004:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2144.260117] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2144.263536] LustreError: 101004:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 [ 2144.273351] Lustre: 101004:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted: req@000000003bcbd0a8 x1846872132055360/t0(8589934776) o35->83863cc7-5188-49a5-8154-477285d0849c@192.168.204.54@tcp:749/0 lens 392/0 e 0 to 0 dl 1761315334 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'' [ 2144.303632] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 2144.306538] Lustre: Skipped 2 previous similar messages [ 2144.310416] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2144.352542] Lustre: lustre-OST0001: deleting orphan objects from 0x0:47 to 0x0:545 [ 2144.357972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:79 to 0x0:577 [ 2154.983842] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.992982] LustreError: Skipped 1 previous similar message [ 2155.006156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2155.009046] Lustre: Skipped 4 previous similar messages [ 2181.928982] LustreError: 95805:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315366 with bad export cookie 8164434960316205384 [ 2181.931880] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.939061] LustreError: 95805:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2189.038817] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:16:12 (1761315372) [ 2195.669494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2195.674775] Lustre: *** cfs_fail_loc=197, val=0*** [ 2212.320603] LustreError: 103041:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 1: rc = -5 [ 2223.647452] LustreError: 103041:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2229.824708] Lustre: *** cfs_fail_loc=197, val=0*** [ 2229.826854] Lustre: Skipped 65 previous similar messages [ 2253.471379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.474566] Lustre: Skipped 2 previous similar messages [ 2258.381350] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2266.657757] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2269.135600] Lustre: server umount lustre-OST0000 complete [ 2269.139625] Lustre: Skipped 4 previous similar messages [ 2278.277404] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:17:40 (1761315460) [ 2301.710462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.370043] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.396510] LustreError: Skipped 52 previous similar messages [ 2306.924493] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2315.296955] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2319.195293] LustreError: 104700:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2319.202914] LustreError: 104700:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2319.210246] LustreError: 104700:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -5 [ 2325.471178] Lustre: 105392:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315503/real 1761315503] req@000000006732c410 x1846872141617600/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315509 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2340.852136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.176209] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2369.480859] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2380.163827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2387.067207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2393.606139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2399.917913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.797722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2406.889900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.049365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2407.077893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2407.161662] Lustre: lustre-MDT0000: new disk, initializing [ 2407.295907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2412.143526] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2422.092298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2422.155573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.611217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2426.679153] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2434.659652] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2439.185453] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2445.137913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2445.209528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.215168] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2448.988366] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2457.602416] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2461.855346] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2462.078112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2465.982871] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2466.247517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2470.372712] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2470.374137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.381786] LustreError: Skipped 2 previous similar messages [ 2470.389760] Lustre: Skipped 6 previous similar messages [ 2470.390558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.390564] Lustre: Skipped 9 previous similar messages [ 2486.873576] LustreError: 111110:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315671 with bad export cookie 8164434960316216017 [ 2486.880471] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.889961] LustreError: 111110:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2496.506974] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:21:19 (1761315679) [ 2506.169060] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2509.661538] Key type lgssc unregistered [ 2510.054554] LNet: 113707:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.074298] LNet: Removed LNI 192.168.204.154@tcp [ 2511.009655] Key type .llcrypt unregistered [ 2511.010983] Key type ._llcrypt unregistered [ 2531.052651] alg: No test for adler32 (adler32-zlib) [ 2531.805405] Key type ._llcrypt registered [ 2531.807085] Key type .llcrypt registered [ 2531.933817] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2544.781421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2545.525958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.948709] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2546.100167] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2546.105182] LNet: Accept secure, port 988 [ 2547.751572] Key type lgssc registered [ 2548.721093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.204103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2563.380172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2569.236125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2575.128847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2580.530749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2586.453457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2586.510760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.839847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2587.861529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2587.935925] Lustre: lustre-MDT0000: new disk, initializing [ 2588.025201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.045296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2592.265648] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2602.209649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2602.309213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2602.320955] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2602.571775] Lustre: lustre-OST0000: new disk, initializing [ 2602.578034] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2602.586309] Lustre: Skipped 1 previous similar message [ 2602.670538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.839954] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2615.474426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.516702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.521550] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2615.630239] Lustre: lustre-OST0001: new disk, initializing [ 2615.632765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2615.712691] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.566548] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2631.651048] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.653542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.658021] Lustre: Skipped 1 previous similar message [ 2633.777230] Lustre: server umount lustre-MDT0000 complete [ 2639.365561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2643.549994] LustreError: 117196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761315828 with bad export cookie 13390466010686440101 [ 2643.556037] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.562119] LustreError: 117196:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2643.665780] Lustre: server umount lustre-OST0000 complete [ 2647.083874] Lustre: server umount lustre-OST0001 complete [ 2655.779921] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 10:23:58 (1761315838) [ 2664.677057] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2671.646456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2672.388335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2676.975081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2677.840736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2682.991284] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:24:25 (1761315865) [ 2693.178207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2695.394432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.530213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2700.003260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.148427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.182314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2724.117967] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2733.154801] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2742.272192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2748.735032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2754.346721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2761.342590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.853774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.926428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.103330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2769.125412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2769.185592] Lustre: lustre-MDT0000: new disk, initializing [ 2769.283672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.311737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2773.286834] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2782.389863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2782.466529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.528878] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2782.571394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2782.574394] Lustre: Skipped 1 previous similar message [ 2782.639285] Lustre: lustre-MDT0001: new disk, initializing [ 2782.717283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.749913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2782.759498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2786.460828] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2794.521848] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2800.298659] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2804.994483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.045317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.049517] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2805.196947] Lustre: lustre-OST0000: new disk, initializing [ 2805.202638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2805.266914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.625532] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2811.489225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2811.494310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2818.130596] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.992580] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2822.135983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2825.798339] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2826.003844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2832.360761] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2832.363065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.367938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2832.367946] Lustre: Skipped 1 previous similar message [ 2832.380826] LustreError: Skipped 1 previous similar message [ 2832.404110] Lustre: Skipped 1 previous similar message [ 2833.870576] Lustre: server umount lustre-OST0000 complete [ 2837.491639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.498619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2837.526389] Lustre: Skipped 1 previous similar message [ 2837.539402] Lustre: Skipped 2 previous similar messages [ 2842.592907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.592907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.652682] Lustre: server umount lustre-MDT0000 complete [ 2847.576943] LustreError: 125737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316032 with bad export cookie 13390466010686441312 [ 2847.587107] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2847.596308] LustreError: 125737:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2855.556815] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:27:18 (1761316038) [ 2861.445147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.761490] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.840708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.829872] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2867.180620] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2872.298898] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.511442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.914714] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2884.754161] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.626139] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2895.100627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.331134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.333679] Lustre: Skipped 1 previous similar message [ 2898.527512] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 2904.976687] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2920.358867] Lustre: *** cfs_fail_loc=159, val=0*** [ 2920.361275] Lustre: Skipped 1 previous similar message [ 2927.676434] Lustre: lustre-MDT0000: Client 0bf64845-def1-44ae-9103-59ea5ef62af8 (at 192.168.204.54@tcp) reconnecting [ 2928.551356] Lustre: *** cfs_fail_loc=159, val=0*** [ 2928.555963] Lustre: Skipped 5 previous similar messages [ 2935.868420] Lustre: lustre-MDT0000: Client 0bf64845-def1-44ae-9103-59ea5ef62af8 (at 192.168.204.54@tcp) reconnecting [ 2939.363299] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.369589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.377638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2942.437295] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2945.388553] Lustre: server umount lustre-OST0000 complete [ 2945.393297] Lustre: Skipped 1 previous similar message [ 2950.626332] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.648893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.681031] Lustre: Skipped 1 previous similar message [ 2950.688191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.690933] Lustre: Skipped 2 previous similar messages [ 2957.800133] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.818260] LustreError: Skipped 1 previous similar message [ 2958.069676] LustreError: 130275:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316142 with bad export cookie 13390466010686442166 [ 2958.083812] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2969.394792] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2972.341797] Key type lgssc unregistered [ 2972.730662] LNet: 134168:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2972.748060] LNet: Removed LNI 192.168.204.154@tcp [ 2973.588956] Key type .llcrypt unregistered [ 2973.598382] Key type ._llcrypt unregistered [ 2994.936393] alg: No test for adler32 (adler32-zlib) [ 2995.692721] Key type ._llcrypt registered [ 2995.701307] Key type .llcrypt registered [ 2995.856520] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2996.584903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2997.055432] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 2997.242439] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2997.249694] LNet: Accept secure, port 988 [ 2998.935171] Key type lgssc registered [ 2999.886761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3005.427795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.661157] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3006.715930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.304734] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3012.069582] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.188754] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.959992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.435909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.082891] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3032.141197] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3036.989139] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3043.156321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.481573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.409742] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3054.567109] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3057.912790] Lustre: lustre-OST0000: deleting orphan objects from 0x0:42 to 0x0:65 [ 3067.228380] Lustre: *** cfs_fail_loc=159, val=0*** [ 3074.619485] Lustre: lustre-MDT0000: Client 36baacd2-3706-46fb-b3d8-b95028ea5913 (at 192.168.204.54@tcp) reconnecting [ 3076.241932] Lustre: *** cfs_fail_loc=159, val=0*** [ 3083.331130] Lustre: lustre-MDT0001: Client 36baacd2-3706-46fb-b3d8-b95028ea5913 (at 192.168.204.54@tcp) reconnecting [ 3084.076438] Lustre: *** cfs_fail_loc=159, val=0*** [ 3084.083535] Lustre: Skipped 6 previous similar messages [ 3091.527316] Lustre: lustre-MDT0001: Client 36baacd2-3706-46fb-b3d8-b95028ea5913 (at 192.168.204.54@tcp) reconnecting [ 3096.712163] Lustre: *** cfs_fail_loc=159, val=0*** [ 3103.295381] Lustre: lustre-MDT0000: Client f922538d-b07d-4729-83f4-28ddbd1e7376 (at 192.168.204.54@tcp) reconnecting [ 3104.327115] Lustre: *** cfs_fail_loc=159, val=0*** [ 3104.331906] Lustre: Skipped 14 previous similar messages [ 3111.466453] Lustre: lustre-MDT0000: Client f922538d-b07d-4729-83f4-28ddbd1e7376 (at 192.168.204.54@tcp) reconnecting [ 3118.550202] Lustre: server umount lustre-OST0000 complete [ 3121.633450] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3121.645253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.656061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.707648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.713298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.727150] Lustre: 134945:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316303/real 1761316303] req@000000003a071d22 x1846873875831488/t0(0) o6->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761316310 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3125.734335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.753808] Lustre: 134945:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3125.769082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3125.773370] Lustre: Skipped 2 previous similar messages [ 3126.819070] Lustre: 134943:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316304/real 1761316304] req@000000009c25aeca x1846873875831808/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761316311 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 3126.849564] Lustre: 134943:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 3127.960065] Lustre: server umount lustre-MDT0000 complete [ 3130.851140] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.865048] LustreError: Skipped 1 previous similar message [ 3131.170303] LustreError: 137585:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316315 with bad export cookie 7080942279698625449 [ 3131.181296] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.184249] LustreError: 137585:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3131.638142] Lustre: server umount lustre-MDT0001 complete [ 3141.874153] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3144.575761] Key type lgssc unregistered [ 3144.895875] LNet: 140024:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.901370] LNet: Removed LNI 192.168.204.154@tcp [ 3145.822371] Key type .llcrypt unregistered [ 3145.824049] Key type ._llcrypt unregistered [ 3166.391887] alg: No test for adler32 (adler32-zlib) [ 3167.145661] Key type ._llcrypt registered [ 3167.147941] Key type .llcrypt registered [ 3167.227647] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3167.925821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3168.419218] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3168.595970] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3168.599878] LNet: Accept secure, port 988 [ 3170.231127] Key type lgssc registered [ 3171.265250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3177.409355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.720393] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3178.765095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.928541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3184.098351] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3187.982043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.303526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3192.107619] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3198.752215] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3203.332612] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3209.252373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.468397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.413848] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3217.948841] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3218.687256] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:97 [ 3228.931604] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:129 [ 3241.113142] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3241.851289] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight in log params [ 3244.485407] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params [ 3249.632893] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.633273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.645261] LustreError: Skipped 1 previous similar message [ 3249.653565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.660930] Lustre: Skipped 1 previous similar message [ 3254.753827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.756775] Lustre: Skipped 1 previous similar message [ 3255.117742] Lustre: server umount lustre-OST0000 complete [ 3259.873786] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3259.874676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.879466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3259.885619] Lustre: Skipped 2 previous similar messages [ 3264.293490] Lustre: server umount lustre-MDT0000 complete [ 3264.993371] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.006154] LustreError: Skipped 1 previous similar message [ 3267.226349] LustreError: 141476:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316451 with bad export cookie 9848767990325724396 [ 3267.231410] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3267.234483] LustreError: 141476:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3267.545535] Lustre: server umount lustre-MDT0001 complete [ 3278.295263] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3280.609586] Key type lgssc unregistered [ 3280.858229] LNet: 145576:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3280.864840] LNet: Removed LNI 192.168.204.154@tcp [ 3281.413200] Key type .llcrypt unregistered [ 3281.416207] Key type ._llcrypt unregistered [ 3301.625582] alg: No test for adler32 (adler32-zlib) [ 3302.380525] Key type ._llcrypt registered [ 3302.382257] Key type .llcrypt registered [ 3302.485673] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3303.524549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.202830] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3304.410442] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3304.415159] LNet: Accept secure, port 988 [ 3306.063774] Key type lgssc registered [ 3307.369646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.498654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.142118] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.235746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.861310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3321.315956] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.432802] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.924953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.468195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3333.879446] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3341.073349] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3346.118175] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3351.951356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.202790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.913524] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3359.439945] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:129 [ 3362.965873] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3365.567346] Lustre: lustre-OST0000: deleting orphan objects from 0x0:85 to 0x0:161 [ 3373.086930] Lustre: *** cfs_fail_loc=159, val=0*** [ 3380.285730] Lustre: lustre-MDT0000: Client 81400187-34ec-47ee-bd5a-03bc9315df89 (at 192.168.204.54@tcp) reconnecting [ 3385.312595] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.317730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.329368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3385.847296] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3385.862725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.877483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.948695] Lustre: server umount lustre-OST0000 complete [ 3396.066266] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3396.077727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.111622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3396.123906] Lustre: Skipped 2 previous similar messages [ 3399.852084] Lustre: server umount lustre-MDT0000 complete [ 3401.711154] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.726694] LustreError: Skipped 1 previous similar message [ 3404.191578] LustreError: 147022:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316588 with bad export cookie 12856934280429120936 [ 3404.195274] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3404.220426] LustreError: 147022:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3404.604831] Lustre: server umount lustre-MDT0001 complete [ 3414.449164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3416.493506] Key type lgssc unregistered [ 3416.710709] LNet: 150807:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3416.718210] LNet: Removed LNI 192.168.204.154@tcp [ 3417.390892] Key type .llcrypt unregistered [ 3417.392286] Key type ._llcrypt unregistered [ 3436.434332] alg: No test for adler32 (adler32-zlib) [ 3437.191621] Key type ._llcrypt registered [ 3437.193977] Key type .llcrypt registered [ 3437.292459] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3438.221138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.693495] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3438.899224] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3438.905141] LNet: Accept secure, port 988 [ 3440.591139] Key type lgssc registered [ 3441.590914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.434504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.889267] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.955965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.814905] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3454.946397] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.066270] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.264096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.590699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3465.273878] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3472.070524] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3477.406840] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3483.338824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.680514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.130930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3495.015952] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:161 [ 3498.862319] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3500.198222] Lustre: lustre-OST0000: deleting orphan objects from 0x0:170 to 0x0:193 [ 3523.345550] Lustre: lustre-MDT0001: evicting 28a97eff-f300-4ea2-9d04-9e56396144ae (at 192.168.204.54@tcp) by administrative request [ 3523.350261] Lustre: Skipped 1 previous similar message [ 3551.712328] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.716730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.730577] LustreError: Skipped 1 previous similar message [ 3551.732719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.742895] Lustre: Skipped 1 previous similar message [ 3556.839309] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.850300] LustreError: Skipped 1 previous similar message [ 3556.997644] Lustre: server umount lustre-OST0000 complete [ 3561.953718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.956877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.965566] Lustre: Skipped 1 previous similar message [ 3561.972762] Lustre: Skipped 2 previous similar messages [ 3566.618039] Lustre: server umount lustre-MDT0000 complete [ 3567.073540] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.079933] LustreError: Skipped 1 previous similar message [ 3570.557699] LustreError: 152251:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316755 with bad export cookie 15181080954142342890 [ 3570.559379] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3570.616052] LustreError: 152251:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3570.946469] Lustre: server umount lustre-MDT0001 complete [ 3582.537765] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3585.572362] Key type lgssc unregistered [ 3585.834298] LNet: 156465:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3585.838645] LNet: Removed LNI 192.168.204.154@tcp [ 3586.607756] Key type .llcrypt unregistered [ 3586.612810] Key type ._llcrypt unregistered [ 3646.221149] alg: No test for adler32 (adler32-zlib) [ 3646.979001] Key type ._llcrypt registered [ 3646.980609] Key type .llcrypt registered [ 3647.098955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3647.986104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3648.269348] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3648.432572] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3648.436558] LNet: Accept secure, port 988 [ 3650.105228] Key type lgssc registered [ 3651.075529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3657.508317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.011981] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.095268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.899800] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3664.355501] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3669.475286] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3670.265980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.754542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3675.247680] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3682.519114] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3689.014688] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3696.225406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3696.715280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.794228] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:72 to 0x280000400:193 [ 3700.865413] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3708.115733] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3708.975729] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 3719.139611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.140129] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.145000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.154854] Lustre: Skipped 1 previous similar message [ 3724.258484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3724.269086] Lustre: Skipped 2 previous similar messages [ 3725.308818] Lustre: server umount lustre-OST0000 complete [ 3729.388097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3729.390738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3729.397809] Lustre: Skipped 1 previous similar message [ 3729.405514] Lustre: Skipped 1 previous similar message [ 3734.496399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3734.965773] Lustre: server umount lustre-MDT0000 complete [ 3738.632122] LustreError: 158983:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761316923 with bad export cookie 7676635922158780791 [ 3738.661687] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.072651] Lustre: server umount lustre-MDT0001 complete [ 3749.890435] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3752.660296] Key type lgssc unregistered [ 3752.866555] LNet: 162665:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.876657] LNet: Removed LNI 192.168.204.154@tcp [ 3753.602238] Key type .llcrypt unregistered [ 3753.604249] Key type ._llcrypt unregistered [ 3775.520742] alg: No test for adler32 (adler32-zlib) [ 3776.288646] Key type ._llcrypt registered [ 3776.290277] Key type .llcrypt registered [ 3776.418930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3777.203837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3777.882788] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3778.105031] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3778.110228] LNet: Accept secure, port 988 [ 3779.751170] Key type lgssc registered [ 3780.739136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3787.785577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3795.387226] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:42:58 (1761316978) [ 3809.517478] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 3817.903787] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3827.128565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3833.631473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3839.101713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3844.427675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3850.508245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3850.582490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.821383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3851.859510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3851.980615] Lustre: lustre-MDT0000: new disk, initializing [ 3852.069975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.089776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3856.021915] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3866.102560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3866.186242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.302140] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3866.348167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3866.351059] Lustre: Skipped 1 previous similar message [ 3866.429181] Lustre: lustre-MDT0001: new disk, initializing [ 3866.525686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.577269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3866.592201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3870.456695] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3878.045288] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3883.077399] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3888.532954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3888.587903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.592937] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3888.840909] Lustre: lustre-OST0000: new disk, initializing [ 3888.846146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3888.923554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.843585] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3894.815506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3894.828511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3900.794716] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3910.115037] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3910.127905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.139641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.750120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3915.752739] Lustre: Skipped 1 previous similar message [ 3920.865106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.870398] Lustre: Skipped 1 previous similar message [ 3922.400081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3922.571827] Lustre: server umount lustre-OST0000 complete [ 3925.985582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.992340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3926.003590] Lustre: Skipped 2 previous similar messages [ 3931.104741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.112116] Lustre: Skipped 2 previous similar messages [ 3931.740230] Lustre: server umount lustre-MDT0000 complete [ 3934.840647] LustreError: 167070:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317119 with bad export cookie 17099491880725082594 [ 3934.843762] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3934.860438] LustreError: 167070:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3935.387177] Lustre: server umount lustre-MDT0001 complete [ 3944.644672] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3946.699467] Key type lgssc unregistered [ 3946.922981] LNet: 171077:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3946.929901] LNet: Removed LNI 192.168.204.154@tcp [ 3947.488770] Key type .llcrypt unregistered [ 3947.490421] Key type ._llcrypt unregistered [ 3965.165072] alg: No test for adler32 (adler32-zlib) [ 3965.917634] Key type ._llcrypt registered [ 3965.919111] Key type .llcrypt registered [ 3966.018097] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3966.589987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3966.972670] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 3967.117955] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3967.122411] LNet: Accept secure, port 988 [ 3968.735154] Key type lgssc registered [ 3969.560560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3975.618572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.968143] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3977.021840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.125404] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3982.307331] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.822725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.201570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3990.321832] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 3996.783233] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4002.680653] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4007.499482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4007.886137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.083311] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4017.050533] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4027.371117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:33 [ 4031.231633] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4031.251708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.256360] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4031.260020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4031.267378] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 4031.270811] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:65 [ 4032.160024] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4032.192798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.216194] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4032.219826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4032.252270] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 4032.253478] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3 to 0x0:97 [ 4033.107229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.113784] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:161 [ 4033.857131] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4033.859543] Lustre: Skipped 1 previous similar message [ 4033.872307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.878679] Lustre: Skipped 1 previous similar message [ 4033.881481] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4033.884453] Lustre: Skipped 1 previous similar message [ 4033.886856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.902929] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 4033.904514] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:193 [ 4033.912301] Lustre: Skipped 1 previous similar message [ 4034.694229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4034.705859] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:225 [ 4035.558722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4035.580708] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:257 [ 4036.432979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4036.443975] Lustre: Skipped 2 previous similar messages [ 4036.459452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.466958] Lustre: Skipped 2 previous similar messages [ 4036.471224] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4036.475153] Lustre: Skipped 2 previous similar messages [ 4036.477927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4036.485725] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 4036.488819] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:289 [ 4036.490199] Lustre: Skipped 2 previous similar messages [ 4037.336347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4037.352475] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:321 [ 4038.277178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4038.293765] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:353 [ 4039.211982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4039.223376] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:385 [ 4047.841149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.841742] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4047.851754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4047.853400] Lustre: Skipped 4 previous similar messages [ 4053.039817] Lustre: server umount lustre-OST0000 complete [ 4058.080308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.081561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4058.089592] Lustre: Skipped 1 previous similar message [ 4058.095393] Lustre: Skipped 1 previous similar message [ 4062.744218] Lustre: server umount lustre-MDT0000 complete [ 4063.203131] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.213697] LustreError: Skipped 1 previous similar message [ 4065.896448] LustreError: 172519:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317250 with bad export cookie 12161274782973562615 [ 4065.910970] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4066.261139] Lustre: server umount lustre-MDT0001 complete [ 4075.694310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4077.875393] Key type lgssc unregistered [ 4078.096638] LNet: 177017:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4078.103987] LNet: Removed LNI 192.168.204.154@tcp [ 4078.759305] Key type .llcrypt unregistered [ 4078.761176] Key type ._llcrypt unregistered [ 4096.015755] alg: No test for adler32 (adler32-zlib) [ 4096.769405] Key type ._llcrypt registered [ 4096.771668] Key type .llcrypt registered [ 4096.871887] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4097.440042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4098.088698] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4098.253403] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4098.258761] LNet: Accept secure, port 988 [ 4099.895106] Key type lgssc registered [ 4100.776806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4106.105450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.414881] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.457134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4110.420109] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4112.864315] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.290920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.546435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4119.656948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4125.622089] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4130.612753] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4137.024819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4137.472183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.522521] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:417 [ 4141.834693] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4148.648307] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4153.495138] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4155.554959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.562936] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4155.569122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.580895] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 4155.583893] LustreError: 178499:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 4160.587221] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 4162.021478] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.021717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.032371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.038418] Lustre: Skipped 1 previous similar message [ 4164.036935] Lustre: server umount lustre-OST0000 complete [ 4175.647141] LustreError: 178499:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake [ 4175.652280] LustreError: 178499:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4198.191286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4198.464888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.508372] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4210.604797] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4214.890396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4214.901403] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4214.903680] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.204.154@tcp (at 0@lo) [ 4214.933551] Lustre: lustre-OST0000: deleting orphan objects from 0x0:122 to 0x0:449 [ 4214.946848] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:66 to 0x280000400:97 [ 4215.439261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4221.721715] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8aab11895000.ost_server_uuid 40 [ 4223.158719] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8aab11895000.ost_server_uuid in FULL state after 0 sec [ 4229.407722] Lustre: server umount lustre-OST0000 complete [ 4234.208163] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4234.213385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.221091] Lustre: Skipped 1 previous similar message [ 4234.226283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.791319] Lustre: 177792:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317415/real 1761317415] req@00000000d50597a3 x1846875030314688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761317422 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [ 4237.798433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.802019] Lustre: 177792:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4237.808875] Lustre: Skipped 1 previous similar message [ 4237.810922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.810928] Lustre: Skipped 2 previous similar messages [ 4238.815216] Lustre: 182966:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317417/real 1761317417] req@000000000bcea568 x1846875030315712/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761317423 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 4239.030184] Lustre: server umount lustre-MDT0000 complete [ 4242.330748] LustreError: 178460:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317426 with bad export cookie 13110895521653788150 [ 4242.342564] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4242.343528] LustreError: 178460:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4242.741729] Lustre: server umount lustre-MDT0001 complete [ 4252.630795] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4255.276413] Key type lgssc unregistered [ 4255.517279] LNet: 183770:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4255.531174] LNet: Removed LNI 192.168.204.154@tcp [ 4256.351370] Key type .llcrypt unregistered [ 4256.352874] Key type ._llcrypt unregistered [ 4284.046578] alg: No test for adler32 (adler32-zlib) [ 4284.809382] Key type ._llcrypt registered [ 4284.810774] Key type .llcrypt registered [ 4284.902911] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4285.837980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4286.641925] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 4286.997307] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4287.007290] LNet: Accept secure, port 988 [ 4288.712187] Key type lgssc registered [ 4289.680494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4300.642681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.466026] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.570203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4306.593097] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.146972] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4312.039024] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.139049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.445752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4319.341021] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4327.517397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.934821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.023959] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 4333.180576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4342.694775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.770743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4342.781358] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4342.899696] Lustre: lustre-OST0001: new disk, initializing [ 4342.904540] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4342.995160] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4343.465610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4343.481167] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4347.771169] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4357.227926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4367.884739] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4389.857842] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.861732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.883793] Lustre: Skipped 2 previous similar messages [ 4389.902647] Lustre: Skipped 1 previous similar message [ 4394.980456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4394.986749] Lustre: Skipped 1 previous similar message [ 4395.293431] Lustre: server umount lustre-MDT0000 complete [ 4398.416785] LustreError: 185528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317582 with bad export cookie 12157218291962040192 [ 4398.418678] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4398.422874] LustreError: 185528:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4398.682390] Lustre: server umount lustre-MDT0001 complete [ 4406.751189] Lustre: 189330:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317585/real 1761317585] req@0000000026dae211 x1846875228517952/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317591 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4406.841681] Lustre: server umount lustre-OST0000 complete [ 4407.205098] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.211673] Lustre: Skipped 1 previous similar message [ 4409.776707] Lustre: server umount lustre-OST0001 complete [ 4412.679250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4412.751780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4413.711608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4413.823506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4414.812279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4414.931074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4415.830479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4415.897663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4426.092838] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4433.744306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.316501] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.443467] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.944944] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4439.522751] LustreError: 137-5: mylustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.126803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.497544] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4449.526709] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4455.919695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.087402] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4464.358158] LustreError: 137-5: mylustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.373985] Lustre: mylustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:513 [ 4464.380480] LustreError: Skipped 2 previous similar messages [ 4465.937910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.074461] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4466.078198] Lustre: Skipped 1 previous similar message [ 4467.939793] Lustre: mylustre-OST0001: deleting orphan objects from 0x0:3 to 0x0:33 [ 4470.194540] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4477.784233] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4481.731706] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4496.864092] LustreError: 11-0: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4496.870534] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.879720] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4496.915776] Lustre: Skipped 3 previous similar messages [ 4501.989963] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4501.992898] Lustre: Skipped 4 previous similar messages [ 4507.107416] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.111615] Lustre: Skipped 4 previous similar messages [ 4510.175128] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4510.264053] Lustre: server umount mylustre-MDT0000 complete [ 4512.226116] LustreError: 137-5: mylustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.242707] LustreError: Skipped 3 previous similar messages [ 4513.442059] LustreError: 194187:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317697 with bad export cookie 12157218291962042915 [ 4513.446505] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4513.451648] LustreError: 194187:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4523.487157] Lustre: 194696:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317701/real 1761317701] req@0000000042ffcad0 x1846875228548032/t0(0) o39->mylustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317707 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 4523.510495] Lustre: 194696:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4523.605152] Lustre: server umount mylustre-OST0000 complete [ 4523.614387] Lustre: Skipped 1 previous similar message [ 4524.449198] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.476864] Lustre: 184813:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317702/real 1761317702] req@00000000288bf2ed x1846875228548096/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317709 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4525.518933] Lustre: 184813:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4530.875543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4531.012632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4532.058745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4532.225882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.352453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4533.446782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4534.456831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4534.611724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4549.192701] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4557.797865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.134976] LustreError: 137-5: tfs-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.194607] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.865921] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4569.812706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.598995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4581.639334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.341461] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4588.034890] Lustre: tfs-OST0000: deleting orphan objects from 0x0:483 to 0x0:545 [ 4592.977612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.474282] Lustre: tfs-OST0001: deleting orphan objects from 0x0:35 to 0x0:65 [ 4596.750315] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4603.686536] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4607.238842] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4618.727583] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.728752] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.735833] Lustre: Skipped 2 previous similar messages [ 4618.748123] Lustre: Skipped 3 previous similar messages [ 4623.514875] Lustre: server umount tfs-MDT0000 complete [ 4623.516632] Lustre: Skipped 1 previous similar message [ 4623.845162] LustreError: 137-5: tfs-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.863721] LustreError: Skipped 8 previous similar messages [ 4626.444282] LustreError: 198697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317810 with bad export cookie 12157218291962044749 [ 4626.452615] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.454719] LustreError: 198697:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4635.999157] Lustre: 184814:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317813/real 1761317813] req@00000000782f4688 x1846875228572480/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317820 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4636.042842] Lustre: 184814:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4636.052363] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.087479] Lustre: Skipped 1 previous similar message [ 4643.316743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4643.448555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4644.349070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 4644.425243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.335922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4645.403294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.218393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 4646.310285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.459459] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4667.204237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.586722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.599090] Lustre: Skipped 3 previous similar messages [ 4671.894794] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4679.449205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.943745] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4691.242905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4695.604597] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4697.717441] Lustre: lustre-OST0000: deleting orphan objects from 0x0:483 to 0x0:577 [ 4703.179541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4705.052489] Lustre: lustre-OST0001: deleting orphan objects from 0x0:35 to 0x0:97 [ 4707.215323] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4713.571340] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4716.545329] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4727.980987] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:58:30 (1761317910) [ 4744.675926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4744.682271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4744.698694] Lustre: Skipped 3 previous similar messages [ 4744.720075] Lustre: Skipped 2 previous similar messages [ 4749.436553] Lustre: server umount lustre-MDT0000 complete [ 4749.440135] Lustre: Skipped 3 previous similar messages [ 4752.638111] LustreError: 201642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317937 with bad export cookie 12157218291962046380 [ 4752.638765] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.649848] LustreError: 201642:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4754.913637] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4754.927298] LustreError: Skipped 13 previous similar messages [ 4782.965707] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4791.462163] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4801.295561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4806.413506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4811.040144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4815.749437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4821.698085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4821.772529] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 4821.772529] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4821.772529] [ 4821.788544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4821.986729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4822.020584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 4822.107398] Lustre: lustre-MDT0000: new disk, initializing [ 4822.211341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4822.214393] Lustre: Skipped 3 previous similar messages [ 4822.240434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4826.243783] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4836.449527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4836.526356] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 4836.526356] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 4836.526356] [ 4836.542558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 4836.666607] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4836.698783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 4836.704801] Lustre: Skipped 1 previous similar message [ 4836.771918] Lustre: lustre-MDT0001: new disk, initializing [ 4836.923139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4836.934499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4841.393818] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4852.139796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4852.209140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.214975] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 4852.462758] Lustre: lustre-OST0000: new disk, initializing [ 4852.465708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 4853.919781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4853.929509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4857.318670] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4867.295801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4867.355519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4867.359121] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 4867.436286] Lustre: lustre-OST0001: new disk, initializing [ 4867.439621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 4869.380894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4869.391822] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4871.524965] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 4882.900918] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:01:05 (1761318065) [ 4895.718225] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:01:18 (1761318078) [ 4910.569982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.572890] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4910.576254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4910.576263] Lustre: Skipped 4 previous similar messages [ 4910.590055] Lustre: Skipped 3 previous similar messages [ 4915.354126] Lustre: server umount lustre-OST0000 complete [ 4915.387188] Lustre: Skipped 3 previous similar messages [ 4933.087175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4936.411385] LustreError: 209622:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318120 with bad export cookie 12157218291962047899 [ 4936.431807] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.712049] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4976.246117] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4987.170084] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4988.739418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4995.629941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5001.522048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5007.744533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5013.898169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5020.267856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5020.327646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.472511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5020.499804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5020.549820] Lustre: lustre-MDT0000: new disk, initializing [ 5020.663975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5024.917576] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5035.163986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5035.213571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.514377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5038.842891] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5046.044943] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5049.955865] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5055.829524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5055.881472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5055.886673] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5056.113499] Lustre: lustre-OST0000: new disk, initializing [ 5056.115367] Lustre: Skipped 1 previous similar message [ 5056.119743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5056.123877] Lustre: Skipped 2 previous similar messages [ 5057.456629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5057.473893] Lustre: Skipped 1 previous similar message [ 5057.475689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5059.941858] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5068.338962] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5077.988219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.990196] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5077.997081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5077.997091] Lustre: Skipped 12 previous similar messages [ 5078.000284] Lustre: Skipped 5 previous similar messages [ 5078.008529] LustreError: Skipped 2 previous similar messages [ 5079.969205] Lustre: server umount lustre-OST0000 complete [ 5079.974744] Lustre: Skipped 3 previous similar messages [ 5086.690105] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5093.341136] LustreError: 215125:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318277 with bad export cookie 12157218291962049327 [ 5093.345168] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.350476] LustreError: 215125:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5106.224261] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5109.039313] Key type lgssc unregistered [ 5109.322632] LNet: 218983:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5109.328102] LNet: Removed LNI 192.168.204.154@tcp [ 5110.150861] Key type .llcrypt unregistered [ 5110.153779] Key type ._llcrypt unregistered [ 5138.628770] alg: No test for adler32 (adler32-zlib) [ 5139.386103] Key type ._llcrypt registered [ 5139.387945] Key type .llcrypt registered [ 5139.480319] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5140.516296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5141.277973] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5141.521625] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5141.526948] LNet: Accept secure, port 988 [ 5143.239631] Key type lgssc registered [ 5144.160330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5151.619064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.998050] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5153.075823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.267316] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5158.371568] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.490984] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.510411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.905518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5168.276436] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5173.619554] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5178.670904] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5184.207872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5184.445985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5188.511403] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5196.390294] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5197.334497] Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 [ 5199.842193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.861032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5201.609277] Lustre: server umount lustre-OST0000 complete [ 5205.985290] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5205.988968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5206.019529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5208.036950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5208.037338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.465163] Lustre: server umount lustre-MDT0000 complete [ 5213.152080] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.176307] LustreError: Skipped 1 previous similar message [ 5214.885047] LustreError: 223121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318399 with bad export cookie 10173335754260576249 [ 5214.889908] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5214.900199] LustreError: 223121:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5215.276182] Lustre: server umount lustre-MDT0001 complete [ 5226.174788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5228.788720] Key type lgssc unregistered [ 5229.082354] LNet: 224230:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5229.096454] LNet: Removed LNI 192.168.204.154@tcp [ 5229.838144] Key type .llcrypt unregistered [ 5229.841767] Key type ._llcrypt unregistered [ 5243.078044] alg: No test for adler32 (adler32-zlib) [ 5243.836366] Key type ._llcrypt registered [ 5243.844390] Key type .llcrypt registered [ 5243.937318] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5245.100040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5245.450614] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5245.648645] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5245.651718] LNet: Accept secure, port 988 [ 5247.335213] Key type lgssc registered [ 5248.192505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5256.966114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5274.527291] LustreError: 225473:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 1: rc = -5 [ 5305.248991] LustreError: 225473:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 5335.008686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.973675] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5345.805446] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5348.193291] Lustre: server umount lustre-OST0000 complete [ 5365.206642] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5368.799285] Key type lgssc unregistered [ 5369.216298] LNet: 227119:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5369.234409] LNet: Removed LNI 192.168.204.154@tcp [ 5370.123669] Key type .llcrypt unregistered [ 5370.127842] Key type ._llcrypt unregistered [ 5402.056961] alg: No test for adler32 (adler32-zlib) [ 5402.823902] Key type ._llcrypt registered [ 5402.829727] Key type .llcrypt registered [ 5402.993805] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5403.762833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5404.188182] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5404.353497] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5404.358792] LNet: Accept secure, port 988 [ 5406.016914] Key type lgssc registered [ 5406.860426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5419.028371] loop0: detected capacity change from 0 to 204800000 [ 5419.234712] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.249507] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.587912] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.594382] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.640442] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.672042] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.727604] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.792607] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.887028] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5419.934902] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5421.117654] loop0: detected capacity change from 0 to 204800000 [ 5421.290692] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5422.100717] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5423.282536] loop0: detected capacity change from 0 to 204800000 [ 5423.670543] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5424.500474] loop0: detected capacity change from 0 to 204800000 [ 5424.697044] print_req_error: 12 callbacks suppressed [ 5424.697057] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5424.709202] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5424.811217] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5424.863454] blk_update_request: operation not supported error, dev loop0, sector 58272 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5425.108583] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5426.223875] loop0: detected capacity change from 0 to 204800000 [ 5426.279309] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5430.568471] loop0: detected capacity change from 0 to 204800000 [ 5430.599916] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5434.726136] loop0: detected capacity change from 0 to 204800000 [ 5434.748350] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5438.628411] loop0: detected capacity change from 0 to 204800000 [ 5438.653688] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 5442.594217] loop0: detected capacity change from 0 to 204800000 [ 5442.678492] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5442.774405] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.827449] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5442.835506] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 5442.844552] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5442.858221] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5442.878839] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5442.886491] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 5442.894780] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 5442.902898] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 5442.919729] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 5442.933607] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 5442.939934] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 5442.945528] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 5442.953521] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 5442.966093] Lustre: 230580:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 5454.815667] Lustre: 230859:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318633/real 1761318633] req@00000000404b921a x1846876399730816/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761318639 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 5454.880233] Lustre: server umount MGS complete [ 5455.741820] loop0: detected capacity change from 0 to 204800000 [ 5455.887835] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.313977] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.386691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.431676] LustreError: 230913:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 5457.316532] loop1: detected capacity change from 0 to 204800000 [ 5457.413530] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5457.542843] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.606277] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5457.613940] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5457.620561] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 5457.626716] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 5457.633720] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 5457.642257] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5457.649599] Lustre: 231028:0:(osd_scrub.c:1764:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5457.662780] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5457.800908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5457.857919] LustreError: 231028:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0001: Aborting recovery for device [ 5458.814892] loop2: detected capacity change from 0 to 204800000 [ 5458.988909] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5459.055957] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5459.062339] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/loop2 [ 5459.094736] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5459.101194] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 5459.110608] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5459.117989] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 5459.124537] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5459.132484] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5459.143394] Lustre: 231111:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 5459.233230] Lustre: Found index 0 for lustre-OST0000, updating log [ 5459.372853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5459.382327] LustreError: 230971:0:(lod_dev.c:424:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000 get update log failed: rc = -108 [ 5459.401042] LustreError: 231111:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0000: aborting recovery [ 5460.344349] loop3: detected capacity change from 0 to 204800000 [ 5460.575124] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5460.694479] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5460.712551] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/loop3 [ 5460.773605] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 5460.783312] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 5460.799540] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 5460.814197] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 5460.828472] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 5460.842277] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 5460.852410] Lustre: 231198:0:(osd_scrub.c:1764:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 5460.875146] Lustre: Found index 1 for lustre-OST0001, updating log [ 5461.028810] LustreError: 231198:0:(ofd_obd.c:1263:ofd_iocontrol()) lustre-OST0001: aborting recovery [ 5462.006248] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6 to 0x0:33 [ 5462.012751] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6 to 0x0:33 [ 5463.035362] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:5 to 0x2c0000400:33 [ 5463.037953] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:5 to 0x280000400:33 [ 5469.144669] LustreError: 231212:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1237us: rc = -2 [ 5469.657680] LustreError: 230935:0:(mdt_identity.c:112:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 831us: rc = -2 [ 5469.690520] LustreError: 230935:0:(mdt_identity.c:112:mdt_identity_do_upcall()) Skipped 26 previous similar messages [ 5483.489367] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.493072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5483.507488] Lustre: Skipped 2 previous similar messages [ 5483.507710] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5483.517359] Lustre: Skipped 3 previous similar messages [ 5487.871540] Lustre: server umount lustre-MDT0000 complete [ 5488.609223] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5488.626152] LustreError: Skipped 3 previous similar messages [ 5488.796329] LustreError: 230920:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318673 with bad export cookie 5968618540475914786 [ 5488.801041] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5488.811680] LustreError: 230920:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5493.729574] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.732600] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.747068] LustreError: Skipped 1 previous similar message [ 5493.750472] Lustre: Skipped 1 previous similar message [ 5493.754537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5493.762571] Lustre: Skipped 1 previous similar message [ 5494.819064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5494.847570] Lustre: Skipped 1 previous similar message [ 5495.200057] Lustre: server umount lustre-MDT0001 complete [ 5502.182250] Lustre: server umount lustre-OST0000 complete [ 5509.424480] Lustre: server umount lustre-OST0001 complete [ 5518.166579] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:11:40 (1761318700) [ 5545.585200] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5553.381342] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5562.503105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5569.514152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5575.564551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5581.668638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.328095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5588.384333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.626702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5588.651727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5588.719389] Lustre: lustre-MDT0000: new disk, initializing [ 5588.786449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.789947] Lustre: Skipped 1 previous similar message [ 5588.811850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5592.914995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5602.167653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5602.261157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.375542] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5602.426356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5602.428453] Lustre: Skipped 1 previous similar message [ 5602.499960] Lustre: lustre-MDT0001: new disk, initializing [ 5602.594494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5602.635812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5602.650829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5606.422463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5615.044778] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5620.289781] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5626.134484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5626.206967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5626.216282] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5626.400730] Lustre: lustre-OST0000: new disk, initializing [ 5626.408051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5626.490542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5630.538750] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5633.086753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5633.094502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5639.745133] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5645.160651] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5657.987524] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 5665.476377] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 5715.425232] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5715.428537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.445546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5715.456210] Lustre: Skipped 2 previous similar messages [ 5717.568370] Lustre: server umount lustre-OST0000 complete [ 5725.667602] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5725.669300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.676312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.687674] Lustre: Skipped 1 previous similar message [ 5725.695538] Lustre: Skipped 1 previous similar message [ 5729.747124] LustreError: 237456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318914 with bad export cookie 5968618540475917131 [ 5729.753355] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5729.761785] LustreError: 237456:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5736.750563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.314124] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5750.239180] Lustre: 240857:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318928/real 1761318928] req@0000000045aab5d8 x1846876399798656/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761318934 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 5750.276610] Lustre: server umount MGS complete [ 5750.281456] Lustre: Skipped 2 previous similar messages [ 5759.819094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.248088] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.262469] LustreError: Skipped 2 previous similar messages [ 5760.273324] LustreError: 241493:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 5760.324277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.108465] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5765.601329] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.389537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.051400] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5781.434739] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5786.853873] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5792.969996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5793.354023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.365608] Lustre: Skipped 1 previous similar message [ 5796.786655] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5802.828257] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5841.892247] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5841.897932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.900548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5841.956140] Lustre: Skipped 1 previous similar message [ 5841.956584] Lustre: Skipped 1 previous similar message [ 5855.202923] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 5855.470908] Lustre: server umount lustre-OST0000 complete [ 5862.372835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.374395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.391960] Lustre: Skipped 1 previous similar message [ 5862.395404] Lustre: Skipped 5 previous similar messages [ 5867.491402] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.507061] LustreError: Skipped 1 previous similar message [ 5868.181738] LustreError: 241462:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319052 with bad export cookie 5968618540475918930 [ 5868.184708] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.202800] LustreError: 241462:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5878.079873] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5880.149227] Key type lgssc unregistered [ 5880.344290] LNet: 245620:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5880.356309] LNet: Removed LNI 192.168.204.154@tcp [ 5880.924723] Key type .llcrypt unregistered [ 5880.927179] Key type ._llcrypt unregistered [ 5912.113994] alg: No test for adler32 (adler32-zlib) [ 5912.870984] Key type ._llcrypt registered [ 5912.876140] Key type .llcrypt registered [ 5912.993530] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 5925.753746] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5926.595501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5927.060258] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 5927.242910] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5927.246549] LNet: Accept secure, port 988 [ 5928.911615] Key type lgssc registered [ 5929.903882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5938.235302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5943.648750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5948.640185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5953.530286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.745382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.796734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.054204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5961.077524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 5961.145394] Lustre: lustre-MDT0000: new disk, initializing [ 5961.214188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.229151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5964.819939] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5973.468738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5973.530478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.625627] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5973.672724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 5973.676024] Lustre: Skipped 1 previous similar message [ 5973.729384] Lustre: lustre-MDT0001: new disk, initializing [ 5973.819250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.858749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5973.866052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5977.067587] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 5984.770836] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5989.518470] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5995.247118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5995.301550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5995.305359] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 5995.456782] Lustre: lustre-OST0000: new disk, initializing [ 5995.462212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 5995.516537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.258930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6004.269095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6004.276153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6006.547260] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.765752] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6014.662509] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff in log lustre-MDT0000 [ 6024.697549] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb in log lustre-client [ 6065.635720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.642490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.642673] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6065.646452] Lustre: Skipped 1 previous similar message [ 6065.661083] Lustre: Skipped 1 previous similar message [ 6070.752108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.758180] Lustre: Skipped 1 previous similar message [ 6071.186977] Lustre: server umount lustre-OST0000 complete [ 6075.873572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.874117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6075.881174] Lustre: Skipped 1 previous similar message [ 6075.888586] Lustre: Skipped 1 previous similar message [ 6079.726239] Lustre: server umount lustre-MDT0000 complete [ 6080.993704] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.001636] LustreError: Skipped 1 previous similar message [ 6082.062116] LustreError: 250551:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319266 with bad export cookie 11894935220988430025 [ 6082.063537] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6082.068567] LustreError: 250551:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6082.301504] Lustre: server umount lustre-MDT0001 complete [ 6087.208502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.924794] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6097.887214] Lustre: 255034:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319276/real 1761319276] req@00000000c2952cb5 x1846876948180672/t0(0) o251->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761319282 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6097.910879] Lustre: server umount MGS complete [ 6106.079526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.382158] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.412134] LustreError: 255670:0:(qmt_pool.c:1415:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 6106.464051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.292674] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6111.714972] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.763034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.977215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.534458] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6122.546398] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6126.133582] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6130.007184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6130.213873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.016563] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6138.064780] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6161.890081] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.895083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.907837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.914776] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6167.691905] Lustre: server umount lustre-OST0000 complete [ 6171.616037] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6171.622729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.630467] Lustre: Skipped 1 previous similar message [ 6171.635975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6171.638641] Lustre: Skipped 2 previous similar messages [ 6176.818290] Lustre: server umount lustre-MDT0000 complete [ 6178.271973] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.292179] LustreError: Skipped 1 previous similar message [ 6179.257497] LustreError: 255638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319363 with bad export cookie 11894935220988431824 [ 6179.258388] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.262885] LustreError: 255638:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6187.020187] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6188.923383] Key type lgssc unregistered [ 6189.213549] LNet: 259773:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6189.219396] LNet: Removed LNI 192.168.204.154@tcp [ 6189.803968] Key type .llcrypt unregistered [ 6189.810247] Key type ._llcrypt unregistered [ 6206.677845] alg: No test for adler32 (adler32-zlib) [ 6207.431402] Key type ._llcrypt registered [ 6207.432696] Key type .llcrypt registered [ 6207.501981] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6208.119409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6208.291267] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 6208.411713] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6208.414591] LNet: Accept secure, port 988 [ 6210.015518] Key type lgssc registered [ 6210.673766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6215.416879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.681712] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.712748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.172646] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6221.792995] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.189943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.384543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.716326] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6231.163374] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6234.074591] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6237.454978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.649729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.838629] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6243.936381] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6247.882102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6247.941230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6247.945135] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 6248.023714] Lustre: lustre-OST0001: new disk, initializing [ 6248.027790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 6248.076977] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6250.576133] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6254.123422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6254.130991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6254.169881] LustreError: 264771:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6254.175472] LustreError: 264771:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6255.202907] LustreError: 264771:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 6255.209186] LustreError: 264771:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6256.224744] LustreError: 264767:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6256.232552] LustreError: 264767:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6257.404243] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6257.524754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6258.274419] LustreError: 264767:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6258.279964] LustreError: 264767:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 6260.756411] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8aab100e6000.ost_server_uuid 40 [ 6261.608344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8aab100e6000.ost_server_uuid in FULL state after 0 sec [ 6262.373402] LustreError: 264767:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 6262.380139] LustreError: 264767:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 6264.863301] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aab100e6000.ost_server_uuid 40 [ 6265.840838] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8aab100e6000.ost_server_uuid in FULL state after 0 sec [ 6279.652912] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.656641] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6279.661877] Lustre: Skipped 1 previous similar message [ 6280.741901] Lustre: lustre-OST0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 6280.744823] Lustre: Skipped 1 previous similar message [ 6284.771374] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6284.775954] Lustre: Skipped 1 previous similar message [ 6284.876161] Lustre: server umount lustre-OST0001 complete [ 6285.844857] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.863842] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6288.868251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.875182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6289.398213] Lustre: server umount lustre-OST0000 complete [ 6293.336829] Lustre: server umount lustre-MDT0000 complete [ 6295.009308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.011500] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.015813] Lustre: Skipped 1 previous similar message [ 6295.236301] LustreError: 261318:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319479 with bad export cookie 11044365806506609819 [ 6295.240200] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6295.242095] LustreError: 261318:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6300.639176] Lustre: 266384:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761319479/real 1761319479] req@00000000ec85a750 x1846877242817920/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761319485 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 6300.843684] Lustre: server umount lustre-MDT0001 complete [ 6306.594287] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6307.833147] Key type lgssc unregistered [ 6307.962410] LNet: 266986:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6307.965530] LNet: Removed LNI 192.168.204.154@tcp [ 6308.317644] Key type .llcrypt unregistered [ 6308.319256] Key type ._llcrypt unregistered [ 6326.923450] alg: No test for adler32 (adler32-zlib) [ 6327.675408] Key type ._llcrypt registered [ 6327.676945] Key type .llcrypt registered [ 6327.725584] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6328.165557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6328.467141] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 6328.585711] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6328.587728] LNet: Accept secure, port 988 [ 6330.199127] Key type lgssc registered [ 6330.801310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6334.690077] loop0: detected capacity change from 0 to 1099511627776 [ 6334.872628] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 6334.876667] *** VALIDATE xfs *** [ 6334.882889] XFS (loop0): Mounting V5 Filesystem [ 6334.895082] XFS (loop0): Ending clean mount [ 6335.525861] loop1: detected capacity change from 0 to 18691697672192 [ 6361.378476] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.679342] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 6373.762950] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6378.320437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.835798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.799513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.861636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6391.488151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6391.524052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.635077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6392.658573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 6392.705470] Lustre: lustre-MDT0000: new disk, initializing [ 6392.755917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.771875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6394.813687] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6400.492596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6400.523951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.574943] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6400.606212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 6400.609287] Lustre: Skipped 1 previous similar message [ 6400.652402] Lustre: lustre-MDT0001: new disk, initializing [ 6400.698636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6400.718334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6400.722723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6402.982867] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6410.987563] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6417.322572] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6424.137439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6424.211346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.222324] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 6424.625776] Lustre: lustre-OST0000: new disk, initializing [ 6424.640097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 6424.758864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.256762] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6430.799937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6430.814826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6441.671537] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6447.672684] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 6448.111782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6454.072493] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 6454.357911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6456.802593] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6456.804753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.834430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6456.839829] Lustre: Skipped 1 previous similar message [ 6461.933922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.042543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.045268] Lustre: Skipped 2 previous similar messages [ 6471.135146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 6471.357407] Lustre: server umount lustre-OST0000 complete [ 6477.286514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.292396] Lustre: Skipped 1 previous similar message [ 6477.297575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6480.649638] Lustre: server umount lustre-MDT0000 complete [ 6482.403951] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.410065] LustreError: Skipped 1 previous similar message [ 6483.521921] LustreError: 271933:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319667 with bad export cookie 7808761758209725786 [ 6483.527677] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.529299] LustreError: 271933:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6483.756297] Lustre: server umount lustre-MDT0001 complete [ 6489.574538] XFS (loop0): Unmounting Filesystem [ 6493.175800] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:27:56 (1761319676) [ 6498.571353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.857176] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.921638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.688499] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6504.421106] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6509.548178] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.995830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.462370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6517.815782] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6525.704389] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6532.018711] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6538.069407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.308092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.446222] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6547.326025] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6559.715655] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6559.719563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.723569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6559.723579] Lustre: Skipped 1 previous similar message [ 6559.734675] LustreError: Skipped 1 previous similar message [ 6559.746018] Lustre: Skipped 2 previous similar messages [ 6565.422825] Lustre: server umount lustre-OST0000 complete [ 6568.930865] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6568.935660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.943431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.949968] Lustre: Skipped 3 previous similar messages [ 6574.418057] Lustre: server umount lustre-MDT0000 complete [ 6575.072817] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.082721] LustreError: Skipped 1 previous similar message [ 6577.881217] LustreError: 276619:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761319762 with bad export cookie 7808761758209726626 [ 6577.882409] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6577.894046] LustreError: 276619:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6589.405953] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6592.672854] Key type lgssc unregistered [ 6592.965681] LNet: 280388:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6592.974344] LNet: Removed LNI 192.168.204.154@tcp [ 6593.941391] Key type .llcrypt unregistered [ 6593.943377] Key type ._llcrypt unregistered [ 6612.858543] alg: No test for adler32 (adler32-zlib) [ 6613.612536] Key type ._llcrypt registered [ 6613.615366] Key type .llcrypt registered [ 6613.677863] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6614.174501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6614.518934] Lustre: Lustre: Build Version: 2.15.7_13_gae8ad20 [ 6614.653900] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6614.665276] LNet: Accept secure, port 988 [ 6616.303228] Key type lgssc registered [ 6617.065549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6623.731843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.097475] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.150245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.011976] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6630.370132] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.400890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.487988] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.684153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6639.016402] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6644.726747] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6648.423953] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6653.513645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.723283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6657.058473] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6663.693036] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6675.501975] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 6675.937075] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6675.949866] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.451976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6676.452522] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.462295] Lustre: Skipped 1 previous similar message [ 6679.354459] Lustre: server umount lustre-MDT0001 complete [ 6680.601134] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6681.278908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6685.715613] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.726785] LustreError: Skipped 2 previous similar messages [ 6686.748832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.840754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6686.843376] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6686.872801] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6687.106550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6690.121413] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 6692.327561] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 6692.331155] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6694.381481] LustreError: 285787:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 6694.387544] LustreError: 285787:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 6694.393653] LustreError: 285787:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 6694.401754] LustreError: 281996:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 6694.437642] LustreError: 285797:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -17 [ 6694.446252] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 6694.446252] [ 6998.051179] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 6998.054775] Lustre: Skipped 1 previous similar message [ 6999.522314] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6999.522971] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7003.155381] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 7003.162068] Lustre: Skipped 2 previous similar messages [ 7008.281590] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 7008.287076] Lustre: Skipped 2 previous similar messages [ 7009.247156] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7009.371666] Lustre: server umount lustre-MDT0001 complete [ 7009.761490] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.768510] LustreError: Skipped 3 previous similar messages [ 7011.267803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7016.127211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7016.196427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7016.199334] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7016.228112] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7016.386511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7018.591601] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 7021.546382] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7021.554077] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 7021.556905] Lustre: Skipped 2 previous similar messages [ 7023.593955] LustreError: 296134:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7023.600181] LustreError: 296134:0:(genops.c:522:class_register_device()) Skipped 1 previous similar message [ 7023.606883] LustreError: 296134:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7023.612888] LustreError: 296134:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7023.619405] LustreError: 281996:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7023.622950] LustreError: 281996:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7023.653906] LustreError: 296137:0:(obd_config.c:680:class_setup()) Device 8 already setup (type osp) [ 7023.657455] LustreError: 296137:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -17 [ 7023.661771] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.154@tcp [ 7023.661771] [ 7323.618639] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.621729] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7323.631106] Lustre: Skipped 1 previous similar message [ 7323.633451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7323.646975] Lustre: Skipped 1 previous similar message [ 7327.421303] Lustre: server umount lustre-MDT0001 complete [ 7328.738186] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.743823] LustreError: Skipped 3 previous similar messages [ 7328.788946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7332.168056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7332.235571] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7332.238388] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7332.260064] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7332.415712] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7334.777531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 7337.445835] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 7337.450328] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7342.057477] LustreError: 307217:0:(genops.c:522:class_register_device()) lustre-MDT0001-lwp-OST0000: already exists, won't add [ 7342.061827] LustreError: 307217:0:(obd_mount.c:195:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 7342.066142] LustreError: 307217:0:(obd_mount_server.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 7342.072431] LustreError: 281996:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 7342.078140] LustreError: 281996:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 7342.112211] LustreError: 307220:0:(lod_lov.c:228:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 7342.119763] LustreError: 307220:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -114 [ 7342.124433] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 7342.124433] [ 7652.835607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.837292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.842895] Lustre: Skipped 3 previous similar messages [ 7652.848571] Lustre: Skipped 3 previous similar messages [ 7665.119182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 7667.244142] Lustre: server umount lustre-MDT0000 complete [ 7668.193609] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.202400] LustreError: Skipped 4 previous similar messages [ 7669.139656] LustreError: 285266:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320853 with bad export cookie 8861864866884481021 [ 7669.142436] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7669.144435] LustreError: 285266:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7673.312460] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.359753] Lustre: server umount lustre-MDT0001 complete [ 7681.603505] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7685.566527] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7690.061436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7693.490952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7696.420852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7699.456055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7702.714537] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:48:06 (1761320886) [ 7706.086130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7706.124265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.252898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7706.269246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 7706.310663] Lustre: lustre-MDT0000: new disk, initializing [ 7706.351328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.361284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7708.218909] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 7713.523039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7713.553034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.595210] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7713.619641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 7713.622795] Lustre: Skipped 1 previous similar message [ 7713.664661] Lustre: lustre-MDT0001: new disk, initializing [ 7713.721931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7713.741222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7713.747343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7715.642838] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 7720.627767] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7723.187851] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7726.161070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7726.188419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7726.192656] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 7726.306845] Lustre: lustre-OST0000: new disk, initializing [ 7726.309665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 7726.346140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7726.619578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7726.623719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7727.983863] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all 8 [ 7732.532767] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7736.800731] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.801749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.806978] Lustre: Skipped 2 previous similar messages [ 7736.810777] Lustre: Skipped 10 previous similar messages [ 7739.951638] Lustre: server umount lustre-MDT0000 complete [ 7739.953221] Lustre: Skipped 1 previous similar message [ 7741.633394] LustreError: 320672:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761320926 with bad export cookie 8861864866884482932 [ 7741.635050] LustreError: 321542:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [ 7741.636144] LustreError: 166-1: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7741.638170] LustreError: 320672:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7741.642171] LustreError: 321542:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 llog-records: rc = -5 [ 7741.656125] LustreError: 321542:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -5 [ 7750.111304] Lustre: 323931:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761320928/real 1761320928] req@00000000ee034b6f x1846877669741312/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761320934 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 7763.141257] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7767.386153] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7772.197069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7775.737150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7778.654832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7781.555370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7782.295863] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7554 sec ========== 11:49:25 (1761320965) [ 7800.154636] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7801.360691] Key type lgssc unregistered [ 7801.497806] LNet: 329029:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7801.501785] LNet: Removed LNI 192.168.204.154@tcp [ 7801.864738] Key type .llcrypt unregistered [ 7801.866818] Key type ._llcrypt unregistered