[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473664657 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002246] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009118] LSM: Security Framework initializing [ 0.010052] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021723] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025598] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030537] *** VALIDATE proc *** [ 0.032219] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034109] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037034] Spectre V2 : User space: Vulnerable [ 0.038019] Speculative Store Bypass: Vulnerable [ 0.040000] debug: unmapping init [mem 0xffffffff8fa59000-0xffffffff8fa60fff] [ 0.042190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044076] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045032] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047009] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.053260] rcu: Hierarchical SRCU implementation. [ 0.055619] smp: Bringing up secondary CPUs ... [ 0.056622] x86: Booting SMP configuration: [ 0.057023] .... node #0, CPUs: #1 #2 #3 [ 0.063147] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.253180] node 0 deferred pages initialised in 185ms [ 0.257141] devtmpfs: initialized [ 0.258242] x86/mm: Memory block size: 128MB [ 0.261000] gcov: version magic: 0x41383552 [ 0.263401] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.270138] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.273435] pinctrl core: initialized pinctrl subsystem [ 0.275864] [ 0.276008] ************************************************************* [ 0.279010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280009] ** ** [ 0.282014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.285019] ** ** [ 0.288018] ** This means that this kernel is built to expose internal ** [ 0.290011] ** IOMMU data structures, which may compromise security on ** [ 0.292015] ** your system. ** [ 0.294013] ** ** [ 0.296011] ** If you see this message and you are not debugging the ** [ 0.298014] ** kernel, report this immediately to your vendor! ** [ 0.299011] ** ** [ 0.302019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304015] ************************************************************* [ 0.307918] NET: Registered protocol family 16 [ 0.309515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.312083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.315174] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320062] cpuidle: using governor menu [ 0.323577] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.327556] PCI: Using configuration type 1 for base access [ 0.329100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.338346] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.341018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.345223] cryptd: max_cpu_qlen set to 1000 [ 0.351248] ACPI: Added _OSI(Module Device) [ 0.352000] ACPI: Added _OSI(Processor Device) [ 0.354031] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.356016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.361000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.370572] ACPI: Interpreter enabled [ 0.371114] ACPI: PM: (supports S0 S3 S4 S5) [ 0.371921] ACPI: Using IOAPIC for interrupt routing [ 0.374100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.376391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.384000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.387049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.393105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.398450] acpiphp: Slot [2] registered [ 0.400163] acpiphp: Slot [5] registered [ 0.401207] acpiphp: Slot [6] registered [ 0.404218] acpiphp: Slot [7] registered [ 0.405154] acpiphp: Slot [8] registered [ 0.407155] acpiphp: Slot [9] registered [ 0.409224] acpiphp: Slot [10] registered [ 0.411134] acpiphp: Slot [3] registered [ 0.412110] acpiphp: Slot [4] registered [ 0.414103] acpiphp: Slot [11] registered [ 0.415123] acpiphp: Slot [12] registered [ 0.416124] acpiphp: Slot [13] registered [ 0.418101] acpiphp: Slot [14] registered [ 0.419080] acpiphp: Slot [15] registered [ 0.421108] acpiphp: Slot [16] registered [ 0.422118] acpiphp: Slot [17] registered [ 0.422950] acpiphp: Slot [18] registered [ 0.424061] acpiphp: Slot [19] registered [ 0.425308] acpiphp: Slot [20] registered [ 0.427102] acpiphp: Slot [21] registered [ 0.429069] acpiphp: Slot [22] registered [ 0.430138] acpiphp: Slot [23] registered [ 0.432100] acpiphp: Slot [24] registered [ 0.433097] acpiphp: Slot [25] registered [ 0.434108] acpiphp: Slot [26] registered [ 0.435199] acpiphp: Slot [27] registered [ 0.437146] acpiphp: Slot [28] registered [ 0.438104] acpiphp: Slot [29] registered [ 0.439058] acpiphp: Slot [30] registered [ 0.441138] acpiphp: Slot [31] registered [ 0.442055] PCI host bridge to bus 0000:00 [ 0.444017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.446025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.449024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.453022] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.455037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460411] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.464464] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.473017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.478173] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.481030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.485027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.487038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.490616] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.493674] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.498055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.503414] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.507016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.521013] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.526019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535162] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.542030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.551031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.563024] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.576919] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.582019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.589024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.608029] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.619000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.625020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.631081] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.648020] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.664000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.671023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.691020] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.706470] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.717000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.721023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.736020] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.745848] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.752025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.758028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.769033] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.785698] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786427] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.787443] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.790696] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.793286] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799035] iommu: Default domain type: Passthrough [ 0.800417] SCSI subsystem initialized [ 0.802198] ACPI: bus type USB registered [ 0.804128] usbcore: registered new interface driver usbfs [ 0.806092] usbcore: registered new interface driver hub [ 0.807082] usbcore: registered new device driver usb [ 0.809198] pps_core: LinuxPPS API ver. 1 registered [ 0.810014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.812075] PTP clock support registered [ 0.813165] EDAC MC: Ver: 3.0.0 [ 0.814468] PCI: Using ACPI for IRQ routing [ 0.817547] NetLabel: Initializing [ 0.819014] NetLabel: domain hash size = 128 [ 0.820010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.822134] NetLabel: unlabeled traffic allowed by default [ 0.825171] vgaarb: loaded [ 0.826000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.826019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.832013] clocksource: Switched to clocksource kvm-clock [ 1.019099] VFS: Disk quotas dquot_6.6.0 [ 1.022868] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.025447] *** VALIDATE ramfs *** [ 1.026742] *** VALIDATE hugetlbfs *** [ 1.029016] pnp: PnP ACPI init [ 1.031495] pnp: PnP ACPI: found 6 devices [ 1.053530] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.056665] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.059375] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.061684] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.064351] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.066933] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.070305] NET: Registered protocol family 2 [ 1.072814] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.076779] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.080311] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.086209] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.089843] TCP: Hash tables configured (established 65536 bind 65536) [ 1.092880] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.096781] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.099571] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.103202] NET: Registered protocol family 1 [ 1.106839] RPC: Registered named UNIX socket transport module. [ 1.109064] RPC: Registered udp transport module. [ 1.111107] RPC: Registered tcp transport module. [ 1.113074] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.116025] NET: Registered protocol family 44 [ 1.117443] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.119423] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.121687] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.124650] PCI: CLS 0 bytes, default 64 [ 1.127410] Unpacking initramfs... [ 2.705720] debug: unmapping init [mem 0xffff9a5c3cc54000-0xffff9a5c3ffbffff] [ 2.709451] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.711414] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.713806] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.267842] Initialise system trusted keyrings [ 3.272775] Key type blacklist registered [ 3.280969] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.292097] zbud: loaded [ 3.298364] *** VALIDATE nfs *** [ 3.299411] *** VALIDATE nfs4 *** [ 3.301198] pstore: using deflate compression [ 3.315255] Platform Keyring initialized [ 3.485189] NET: Registered protocol family 38 [ 3.486693] Key type asymmetric registered [ 3.488181] Asymmetric key parser 'x509' registered [ 3.490232] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.496651] io scheduler mq-deadline registered [ 3.498310] io scheduler kyber registered [ 3.500071] io scheduler bfq registered [ 3.504375] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.512470] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.516181] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.519342] ACPI: Power Button [PWRF] [ 3.632983] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.732734] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.963354] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.067417] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.280779] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.327445] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.380139] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.396126] Non-volatile memory driver v1.3 [ 4.398593] Linux agpgart interface v0.103 [ 4.450782] virtio_blk virtio1: [vda] 67992 512-byte logical blocks (34.8 MB/33.2 MiB) [ 4.454938] vda: detected capacity change from 0 to 34811904 [ 4.486496] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.491908] vdb: detected capacity change from 0 to 1073741824 [ 4.523775] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.527569] vdc: detected capacity change from 0 to 2621440000 [ 4.568649] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.571624] vdd: detected capacity change from 0 to 2621440000 [ 4.598751] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.603259] vde: detected capacity change from 0 to 4294967296 [ 4.627911] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.630381] vdf: detected capacity change from 0 to 4294967296 [ 4.639473] libphy: Fixed MDIO Bus: probed [ 4.665033] usbcore: registered new interface driver usbserial_generic [ 4.669172] usbserial: USB Serial support registered for generic [ 4.672424] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.682574] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.684094] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.687338] mousedev: PS/2 mouse device common for all mice [ 4.690313] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.696908] rtc_cmos 00:05: RTC can wake from S4 [ 4.702356] rtc_cmos 00:05: registered as rtc0 [ 4.706385] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.711620] intel_pstate: CPU model not supported [ 4.712618] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.722125] hid: raw HID events driver (C) Jiri Kosina [ 4.730759] usbcore: registered new interface driver usbhid [ 4.732652] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.736096] usbhid: USB HID core driver [ 4.736386] drop_monitor: Initializing network drop monitor service [ 4.736517] Initializing XFRM netlink socket [ 4.736897] NET: Registered protocol family 10 [ 4.738076] Segment Routing with IPv6 [ 4.738110] NET: Registered protocol family 17 [ 4.738209] mpls_gso: MPLS GSO support [ 4.759580] RAS: Correctable Errors collector initialized. [ 4.760982] AVX version of gcm_enc/dec engaged. [ 4.762440] AES CTR mode by8 optimization enabled [ 4.885604] sched_clock: Marking stable (4885591192, 0)->(5833367865, -947776673) [ 4.889164] registered taskstats version 1 [ 4.890903] Loading compiled-in X.509 certificates [ 4.892764] zswap: loaded using pool lzo/zbud [ 4.922979] Key type big_key registered [ 4.938501] Key type encrypted registered [ 4.940310] ima: No TPM chip found, activating TPM-bypass! [ 4.941933] ima: Allocated hash algorithm: sha1 [ 4.943436] ima: No architecture policies found [ 4.944884] evm: Initialising EVM extended attributes: [ 4.946634] evm: security.selinux [ 4.947650] evm: security.ima [ 4.948541] evm: security.capability [ 4.949604] evm: HMAC attrs: 0x1 [ 4.954534] rtc_cmos 00:05: setting system clock to 2025-10-24 16:02:48 UTC (1761321768) [ 4.961725] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 4.964465] debug: unmapping init [mem 0xffffffff8f782000-0xffffffff8fa58fff] [ 4.974135] Write protecting the kernel read-only data: 28672k [ 4.977339] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 4.980132] debug: unmapping init [mem 0xffffffff8e714000-0xffffffff8e7fffff] [ 5.022870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.031431] systemd[1]: Detected virtualization kvm. [ 5.033372] systemd[1]: Detected architecture x86-64. [ 5.034971] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.063143] systemd[1]: No hostname configured. [ 5.064931] systemd[1]: Set hostname to . [ 5.067251] random: systemd: uninitialized urandom read (16 bytes read) [ 5.069737] systemd[1]: Initializing machine ID from random generator. [ 5.371803] random: systemd: uninitialized urandom read (16 bytes read) [ 5.374855] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.379136] random: systemd: uninitialized urandom read (16 bytes read) [ 5.383741] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.398058] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.419637] device-mapper: uevent: version 1.0.3 [ 6.421565] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.629695] virtio_net virtio0 ens2: renamed from eth0 [ 7.929142] random: fast init done [ 8.315519] scsi host0: ata_piix [ 8.460515] scsi host1: ata_piix [ 8.462198] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.479200] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.464651] random: crng init done [ 12.469622] random: 7 urandom warning(s) missed due to ratelimiting [ 13.524630] dracut-initqueue[578]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.540599] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.195358] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.624470] SELinux: Disabled at runtime. [ 17.709557] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.718570] systemd[1]: Detected virtualization kvm. [ 17.720837] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.701661] hrtimer: interrupt took 7653381 ns [ 18.735534] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.739485] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.746569] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.752239] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.759132] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.772024] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.797578] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ 19.361059] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.951502] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.949828] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.109503] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.888110] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.921751] EDAC sbridge: Ver: 1.1.2 [ 25.345554] Key type dns_resolver registered [ 25.793922] NFS: Registering the id_resolver key type [ 25.796199] Key type id_resolver registered [ 25.798509] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-server login: [ 52.712477] libcfs: loading out-of-tree module taints kernel. [ 52.729465] alg: No test for adler32 (adler32-zlib) [ 53.480378] Key type ._llcrypt registered [ 53.482540] Key type .llcrypt registered [ 53.534594] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 61.710732] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 62.369070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.632440] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 62.956365] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 62.960027] LNet: Accept secure, port 988 [ 64.607547] Key type lgssc registered [ 65.289756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.143643] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.713210] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.501950] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.905532] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.622711] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.956239] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 99.560833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.605905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.723737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 100.738323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 100.779151] Lustre: lustre-MDT0000: new disk, initializing [ 100.818382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.829392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 102.690880] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 108.739060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.785099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 108.829358] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 108.851546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 108.854887] Lustre: Skipped 1 previous similar message [ 108.909176] Lustre: lustre-MDT0001: new disk, initializing [ 108.941679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.955125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 108.959144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 110.853746] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 116.698645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.729204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.733269] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 116.841835] Lustre: lustre-OST0000: new disk, initializing [ 116.846318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 116.867961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.496900] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 121.838150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 121.841447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 124.615785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.647052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 124.649643] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 124.688996] Lustre: lustre-OST0001: new disk, initializing [ 124.690643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 124.711452] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 126.453420] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 129.521925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 129.525179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 132.693871] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.186460] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 143.747092] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.674952] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing yml_node [ 147.750199] Lustre: DEBUG MARKER: Client: 2.15.7.12 [ 148.922691] Lustre: DEBUG MARKER: MDS: 2.15.7.12 [ 150.185129] Lustre: DEBUG MARKER: OSS: 2.15.7.12 [ 150.978840] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 24 12:05:13 EDT 2025 [ 154.634400] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball 110 [ 155.371996] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 [ 170.464114] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 170.464189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 170.469760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 170.474705] Lustre: Skipped 3 previous similar messages [ 170.477085] Lustre: Skipped 2 previous similar messages [ 173.773640] Lustre: server umount lustre-MDT0000 complete [ 175.358044] LustreError: 6210:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761321938 with bad export cookie 12516555404914424075 [ 175.359162] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 175.363015] LustreError: 6210:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 175.584707] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 175.585302] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 175.585680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 175.595572] LustreError: Skipped 1 previous similar message [ 175.604934] Lustre: Skipped 1 previous similar message [ 180.640489] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 180.640676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 180.648175] LustreError: Skipped 1 previous similar message [ 180.650554] Lustre: Skipped 2 previous similar messages [ 181.604157] Lustre: server umount lustre-MDT0001 complete [ 183.496749] Lustre: server umount lustre-OST0000 complete [ 188.272727] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 191.822471] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 196.441570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.572826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.599172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.430418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.703196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.742399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.859223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 208.875753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 208.921934] Lustre: lustre-MDT0000: new disk, initializing [ 208.962199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.972368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.933310] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 216.040268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 216.069824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.112296] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 216.244257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 217.993378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 222.665793] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 225.118528] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 228.369958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 228.411314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.416078] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 228.531632] Lustre: lustre-OST0000: new disk, initializing [ 228.533532] Lustre: Skipped 1 previous similar message [ 228.537556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 228.542543] Lustre: Skipped 2 previous similar messages [ 228.581686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.584605] Lustre: Skipped 1 previous similar message [ 230.339441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.343300] Lustre: Skipped 1 previous similar message [ 230.345411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.457374] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 235.440604] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 237.572640] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 237.678122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 239.820877] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 239.931345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 245.728614] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 245.730968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.732186] LustreError: Skipped 1 previous similar message [ 245.733157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 245.738571] Lustre: Skipped 1 previous similar message [ 247.032335] Lustre: server umount lustre-OST0000 complete [ 247.034299] Lustre: Skipped 1 previous similar message [ 250.849237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.851918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 250.856692] Lustre: Skipped 1 previous similar message [ 250.859725] Lustre: Skipped 1 previous similar message [ 254.741827] Lustre: server umount lustre-MDT0000 complete [ 255.969120] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.975259] LustreError: Skipped 1 previous similar message [ 256.504321] LustreError: 14292:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322020 with bad export cookie 12516555404914425531 [ 256.507192] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.513199] LustreError: 14292:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 258.310220] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:07:00 (1761322020) [ 261.167039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 261.372540] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.407689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.267907] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 266.721757] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 266.801519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 268.856127] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 272.106268] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 274.710931] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 277.778323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 279.964985] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 283.603884] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 297.440483] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 297.443980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.451681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 297.453606] Lustre: Skipped 1 previous similar message [ 299.314185] Lustre: server umount lustre-OST0000 complete [ 299.316200] Lustre: Skipped 1 previous similar message [ 303.072416] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 303.076974] LustreError: Skipped 1 previous similar message [ 313.824098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.826672] Lustre: Skipped 7 previous similar messages [ 315.359148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 315.403520] Lustre: server umount lustre-MDT0000 complete [ 316.968975] LustreError: 19486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322080 with bad export cookie 12516555404914426385 [ 316.971783] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.972387] LustreError: 19486:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.739493] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 323.897065] Key type lgssc unregistered [ 324.036457] LNet: 22374:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.040117] LNet: Removed LNI 192.168.201.121@tcp [ 324.430287] Key type .llcrypt unregistered [ 324.431837] Key type ._llcrypt unregistered [ 334.644895] alg: No test for adler32 (adler32-zlib) [ 335.397420] Key type ._llcrypt registered [ 335.398757] Key type .llcrypt registered [ 335.447814] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 335.953108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 336.124309] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 336.219733] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 336.222193] LNet: Accept secure, port 988 [ 337.815220] Key type lgssc registered [ 338.283793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 341.450101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.638368] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.654068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 344.294296] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 347.610573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.766669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 349.456678] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 352.383444] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 354.720952] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 357.408587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 357.550444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.569350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:35 to 0x0:65 [ 359.234185] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 362.245660] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 368.098549] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 368.102219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.107797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 373.217531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 373.220301] Lustre: Skipped 2 previous similar messages [ 373.673032] Lustre: server umount lustre-OST0000 complete [ 378.337442] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.338487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.348035] Lustre: Skipped 1 previous similar message [ 378.351808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.354284] Lustre: Skipped 1 previous similar message [ 381.326187] Lustre: server umount lustre-MDT0000 complete [ 383.045345] LustreError: 23816:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322146 with bad export cookie 17759637375683098545 [ 383.048262] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.053212] LustreError: 23816:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 383.209152] Lustre: server umount lustre-MDT0001 complete [ 388.416021] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 389.576654] Key type lgssc unregistered [ 389.710794] LNet: 27832:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.714104] LNet: Removed LNI 192.168.201.121@tcp [ 390.043961] Key type .llcrypt unregistered [ 390.046100] Key type ._llcrypt unregistered [ 400.453686] alg: No test for adler32 (adler32-zlib) [ 401.207422] Key type ._llcrypt registered [ 401.208594] Key type .llcrypt registered [ 401.249532] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 401.635041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 401.876332] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 401.968460] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 401.971755] LNet: Accept secure, port 988 [ 403.567198] Key type lgssc registered [ 404.015581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 407.182707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.367683] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.390518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.072666] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 413.541331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.686860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.435709] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 418.558689] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 420.857989] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 426.388133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 426.548954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.429860] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 431.440705] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 438.822984] Lustre: lustre-OST0000: deleting orphan objects from 0x0:67 to 0x0:97 [ 443.874850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.875731] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 443.879990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 443.883059] LustreError: Skipped 1 previous similar message [ 445.677568] Lustre: server umount lustre-OST0000 complete [ 448.992844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.993417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.997795] Lustre: Skipped 2 previous similar messages [ 449.000421] Lustre: Skipped 1 previous similar message [ 449.161546] Lustre: server umount lustre-MDT0000 complete [ 450.624873] LustreError: 29269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322214 with bad export cookie 4744161955029443748 [ 450.626907] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.628789] LustreError: 29269:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 450.747772] Lustre: server umount lustre-MDT0001 complete [ 455.629149] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 456.734370] Key type lgssc unregistered [ 456.862839] LNet: 33303:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.866572] LNet: Removed LNI 192.168.201.121@tcp [ 457.199321] Key type .llcrypt unregistered [ 457.200719] Key type ._llcrypt unregistered [ 466.701544] alg: No test for adler32 (adler32-zlib) [ 467.453477] Key type ._llcrypt registered [ 467.454871] Key type .llcrypt registered [ 467.506296] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 467.899284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 468.155498] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 468.261537] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 468.264416] LNet: Accept secure, port 988 [ 469.863169] Key type lgssc registered [ 470.270318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 473.494222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.685860] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.704317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.567505] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 479.712986] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.587232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.776037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.663562] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 485.708666] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 487.868292] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 490.427936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.570475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.334185] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 495.447335] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 504.934074] Lustre: lustre-OST0000: deleting orphan objects from 0x0:99 to 0x0:129 [ 510.433514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.433614] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.438953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.443884] Lustre: Skipped 1 previous similar message [ 511.145488] Lustre: server umount lustre-OST0000 complete [ 515.552407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.553327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.556967] Lustre: Skipped 2 previous similar messages [ 515.561367] Lustre: Skipped 1 previous similar message [ 518.664467] Lustre: server umount lustre-MDT0000 complete [ 520.130112] LustreError: 36719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322283 with bad export cookie 13774631400879634197 [ 520.132948] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.135140] LustreError: 36719:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 520.277545] Lustre: server umount lustre-MDT0001 complete [ 525.648988] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 526.916437] Key type lgssc unregistered [ 527.048479] LNet: 38443:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.053255] LNet: Removed LNI 192.168.201.121@tcp [ 527.400935] Key type .llcrypt unregistered [ 527.402534] Key type ._llcrypt unregistered [ 537.395991] alg: No test for adler32 (adler32-zlib) [ 538.149411] Key type ._llcrypt registered [ 538.150914] Key type .llcrypt registered [ 538.192325] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 538.678627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 538.829090] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 538.922372] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 538.924728] LNet: Accept secure, port 988 [ 540.511099] Key type lgssc registered [ 540.923127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 543.780345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.972136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.999778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.651199] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 549.911818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.099510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 551.824143] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 554.542660] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 556.854352] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 559.705507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 559.857212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.601706] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 564.630413] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 575.205645] Lustre: lustre-OST0000: deleting orphan objects from 0x0:131 to 0x0:161 [ 580.358088] Lustre: lustre-OST0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 580.578257] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 580.578741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.586641] Lustre: Skipped 1 previous similar message [ 583.719161] Lustre: server umount lustre-OST0000 complete [ 601.568882] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 601.572704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.579393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.582095] Lustre: Skipped 2 previous similar messages [ 603.788100] Lustre: server umount lustre-MDT0000 complete [ 605.345569] LustreError: 41852:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322368 with bad export cookie 11439040704721118356 [ 605.349966] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.351304] LustreError: 41852:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.533737] Lustre: server umount lustre-MDT0001 complete [ 616.729764] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 617.827845] Key type lgssc unregistered [ 617.957380] LNet: 43751:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.961638] LNet: Removed LNI 192.168.201.121@tcp [ 618.296525] Key type .llcrypt unregistered [ 618.298770] Key type ._llcrypt unregistered [ 627.804663] alg: No test for adler32 (adler32-zlib) [ 628.556439] Key type ._llcrypt registered [ 628.557570] Key type .llcrypt registered [ 628.598601] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 629.060073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.210099] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 629.303369] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 629.306390] LNet: Accept secure, port 988 [ 630.887688] Key type lgssc registered [ 631.287523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 634.296742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.479694] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.496947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.149036] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 640.583068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.742631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 642.381803] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 645.132390] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 647.250118] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 649.727365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 649.846848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.855101] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 651.456848] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 654.234761] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 665.863846] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 666.593691] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.593962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.595773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 666.605818] Lustre: Skipped 1 previous similar message [ 667.726810] Lustre: server umount lustre-MDT0000 complete [ 669.264098] LustreError: 45199:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322432 with bad export cookie 5142838080727833002 [ 669.266605] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 669.268163] LustreError: 45199:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 670.973587] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.985272] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.21@tcp (stopping) [ 670.988325] Lustre: Skipped 2 previous similar messages [ 671.712302] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.717544] Lustre: Skipped 1 previous similar message [ 674.719870] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.720055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 674.724991] LustreError: Skipped 1 previous similar message [ 674.729023] Lustre: Skipped 1 previous similar message [ 675.479329] Lustre: server umount lustre-MDT0001 complete [ 693.556960] Lustre: server umount lustre-OST0000 complete [ 700.688152] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 701.742828] Key type lgssc unregistered [ 701.860709] LNet: 49197:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.865710] LNet: Removed LNI 192.168.201.121@tcp [ 702.176428] Key type .llcrypt unregistered [ 702.177899] Key type ._llcrypt unregistered [ 712.464096] alg: No test for adler32 (adler32-zlib) [ 713.215420] Key type ._llcrypt registered [ 713.216988] Key type .llcrypt registered [ 713.264542] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 713.579891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 713.851843] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 713.937984] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 713.940124] LNet: Accept secure, port 988 [ 715.519128] Key type lgssc registered [ 715.894854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 718.756912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 735.263375] LustreError: 50631:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 767.007257] LustreError: 50631:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 797.727298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.362426] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 802.291849] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 840.483723] Lustre: server umount lustre-OST0000 complete [ 848.054974] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 849.207103] Key type lgssc unregistered [ 849.338405] LNet: 52298:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.342125] LNet: Removed LNI 192.168.201.121@tcp [ 849.691510] Key type .llcrypt unregistered [ 849.693316] Key type ._llcrypt unregistered [ 859.391142] alg: No test for adler32 (adler32-zlib) [ 860.143463] Key type ._llcrypt registered [ 860.144721] Key type .llcrypt registered [ 860.189803] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 860.570116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.818588] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 860.917096] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 860.919874] LNet: Accept secure, port 988 [ 862.511181] Key type lgssc registered [ 862.926475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 866.002639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.176752] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.195616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.717245] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 871.949165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.104319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.651424] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 876.408528] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 878.536801] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 881.041040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 881.162232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.698294] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 883.240555] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 885.375791] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 887.328851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.329778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.328126] Lustre: server umount lustre-OST0000 complete [ 897.504500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.506391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.509150] Lustre: Skipped 1 previous similar message [ 897.511483] Lustre: Skipped 1 previous similar message [ 899.851508] Lustre: server umount lustre-MDT0000 complete [ 901.428271] LustreError: 55717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322664 with bad export cookie 7449432540680381392 [ 901.430354] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.433098] LustreError: 55717:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 901.556705] Lustre: server umount lustre-MDT0001 complete [ 906.740128] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 907.821579] Key type lgssc unregistered [ 907.941370] LNet: 57417:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.944938] LNet: Removed LNI 192.168.201.121@tcp [ 908.253202] Key type .llcrypt unregistered [ 908.254616] Key type ._llcrypt unregistered [ 917.747890] alg: No test for adler32 (adler32-zlib) [ 918.500795] Key type ._llcrypt registered [ 918.501980] Key type .llcrypt registered [ 918.541765] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 918.967659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 919.181220] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 919.266527] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 919.268975] LNet: Accept secure, port 988 [ 920.855102] Key type lgssc registered [ 921.305620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.536360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 941.023393] LustreError: 58852:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 972.767290] LustreError: 58852:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 1002.463157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.006196] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1006.721169] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1009.182151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.094881] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.118493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.138994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 1026.541979] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1029.427669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.544524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.969640] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1033.428132] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.373535] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1039.843524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.850615] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.865203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.873852] Lustre: Skipped 1 previous similar message [ 1042.250806] Lustre: server umount lustre-OST0000 complete [ 1065.440445] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.457366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.489542] Lustre: Skipped 1 previous similar message [ 1065.505581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1066.466242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.644585] Lustre: server umount lustre-MDT0000 complete [ 1071.584431] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.595080] LustreError: Skipped 1 previous similar message [ 1072.612141] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.623710] LustreError: Skipped 1 previous similar message [ 1073.209724] LustreError: 61537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322836 with bad export cookie 7802076347587274441 [ 1073.210109] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1073.228705] LustreError: 61537:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1079.992511] Lustre: server umount lustre-MDT0001 complete [ 1095.303780] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1099.104094] Key type lgssc unregistered [ 1099.531988] LNet: 62721:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.555311] LNet: Removed LNI 192.168.201.121@tcp [ 1100.733229] Key type .llcrypt unregistered [ 1100.734911] Key type ._llcrypt unregistered [ 1119.089244] alg: No test for adler32 (adler32-zlib) [ 1119.842501] Key type ._llcrypt registered [ 1119.844120] Key type .llcrypt registered [ 1119.927482] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1120.628236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.939823] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1121.089891] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1121.093674] LNet: Accept secure, port 988 [ 1122.727172] Key type lgssc registered [ 1123.642036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.717523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.963495] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.997923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.483207] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1135.074894] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.553785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.759415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.420256] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1146.028809] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1149.567862] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1153.580187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1153.808106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.503916] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1159.977276] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 1161.263989] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1180.645220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.646973] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1180.654099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1180.655105] LustreError: Skipped 1 previous similar message [ 1180.662866] Lustre: Skipped 1 previous similar message [ 1181.799594] Lustre: server umount lustre-OST0000 complete [ 1185.760702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1185.762368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.765419] Lustre: Skipped 2 previous similar messages [ 1190.720844] Lustre: server umount lustre-MDT0000 complete [ 1190.884035] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.030515] LustreError: 67056:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761322956 with bad export cookie 9615714829603645735 [ 1193.034778] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1193.045930] LustreError: 67056:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1193.268899] Lustre: server umount lustre-MDT0001 complete [ 1201.001545] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1202.578694] Key type lgssc unregistered [ 1202.759222] LNet: 67858:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.764796] LNet: Removed LNI 192.168.201.121@tcp [ 1203.264074] Key type .llcrypt unregistered [ 1203.266389] Key type ._llcrypt unregistered [ 1223.219076] alg: No test for adler32 (adler32-zlib) [ 1223.973527] Key type ._llcrypt registered [ 1223.975087] Key type .llcrypt registered [ 1224.044174] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1224.598935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1224.771795] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1224.892767] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1224.895410] LNet: Accept secure, port 988 [ 1226.503495] Key type lgssc registered [ 1226.990074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1231.187534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.443781] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.488168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.842547] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1237.986458] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.235604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.492115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1245.172413] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1250.385447] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1253.823850] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.657880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.910145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.663697] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1263.024439] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:321 [ 1265.795135] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1269.981590] Lustre: Failing over lustre-MDT0000 [ 1270.091889] Lustre: server umount lustre-MDT0000 complete [ 1272.588372] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.316453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.316742] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1275.876796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.963213] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.965037] LustreError: 11-0: MGC192.168.201.121@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1275.965479] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.966824] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6cfdb47acaff2f5b to 0x6cfdb47acaff333d [ 1275.968838] LustreError: Skipped 5 previous similar messages [ 1275.974070] Lustre: MGC192.168.201.121@tcp: Connection restored to (at 0@lo) [ 1276.057182] Lustre: *** cfs_fail_loc=135, val=0*** [ 1276.124809] BUG: spinlock bad magic on CPU#2, llog_process_th/72766 [ 1276.133870] lock: 0xffff9a5cbc4880c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1276.136589] CPU: 2 PID: 72766 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1276.139876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 1276.141830] Call Trace: [ 1276.143755] ? dump_stack+0xbb/0x10e [ 1276.144685] ? spin_dump.cold.3+0x32/0x65 [ 1276.145715] ? do_raw_spin_lock+0xce/0x190 [ 1276.150147] ? _raw_spin_lock+0x21/0x30 [ 1276.152268] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 1276.154587] ? mdd_process_config+0x368/0x9c0 [mdd] [ 1276.159183] ? mdt_stack_fini+0x5de/0x1200 [mdt] [ 1276.162330] ? mdt_init0+0x650/0x1750 [mdt] [ 1276.164368] ? mdt_device_alloc+0x118/0x170 [mdt] [ 1276.166196] ? obd_setup+0x196/0x3e0 [obdclass] [ 1276.167642] ? class_setup+0x6ee/0x9e0 [obdclass] [ 1276.170001] ? class_process_config+0x17c7/0x2b90 [obdclass] [ 1276.172831] ? class_config_llog_handler+0xb73/0x1ab0 [obdclass] [ 1276.174981] ? llog_process_thread+0x109e/0x22a0 [obdclass] [ 1276.176651] ? keys_fill+0x10d/0x1b0 [obdclass] [ 1276.178592] ? lu_context_init+0xc8/0x240 [obdclass] [ 1276.180383] ? llog_process+0x40/0x40 [obdclass] [ 1276.182000] ? llog_process_thread_daemonize+0xe1/0x150 [obdclass] [ 1276.184479] ? kthread+0x1d1/0x200 [ 1276.185812] ? set_kthread_struct+0x70/0x70 [ 1276.187639] ? ret_from_fork+0x1f/0x30 [ 1276.201168] LustreError: 72766:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 1276.204049] LustreError: 72766:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -2 [ 1276.210849] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1276.210849] [ 1276.214852] LustreError: 15c-8: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1276.220199] LustreError: 72754:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1276.226364] LustreError: 72754:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 1276.235290] LustreError: 72754:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1276.254079] Lustre: server umount lustre-MDT0000 complete [ 1276.257486] LustreError: 72754:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1280.311234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.404585] LustreError: 11-0: MGC192.168.201.121@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1280.405712] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.407287] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.407295] LustreError: Skipped 2 previous similar messages [ 1280.420622] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x6cfdb47acaff333d to 0x6cfdb47acaff33c9 [ 1280.430036] Lustre: MGC192.168.201.121@tcp: Connection restored to (at 0@lo) [ 1280.565390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.605599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1283.249978] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1284.863954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1285.609806] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1285.620092] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1285.650811] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:353 [ 1289.701492] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.706236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.716020] Lustre: Skipped 2 previous similar messages [ 1289.719166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.720322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.723621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.737332] Lustre: Skipped 1 previous similar message [ 1294.613381] Lustre: server umount lustre-OST0000 complete [ 1300.960742] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1300.963197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.966511] LustreError: Skipped 1 previous similar message [ 1300.967777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.972042] Lustre: Skipped 1 previous similar message [ 1300.982746] Lustre: Skipped 1 previous similar message [ 1306.084577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.200375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1311.200844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.215083] Lustre: Skipped 2 previous similar messages [ 1311.283468] Lustre: server umount lustre-MDT0000 complete [ 1312.224536] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.231773] LustreError: Skipped 3 previous similar messages [ 1313.478924] LustreError: 69592:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323077 with bad export cookie 7853631764688745417 [ 1313.481676] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.487923] LustreError: 69592:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1320.759705] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1322.417063] Key type lgssc unregistered [ 1322.571760] LNet: 74648:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.577675] LNet: Removed LNI 192.168.201.121@tcp [ 1323.040227] Key type .llcrypt unregistered [ 1323.042179] Key type ._llcrypt unregistered [ 1337.233221] alg: No test for adler32 (adler32-zlib) [ 1337.986405] Key type ._llcrypt registered [ 1337.988288] Key type .llcrypt registered [ 1338.081378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1338.614462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.944238] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1339.070735] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1339.075141] LNet: Accept secure, port 988 [ 1340.703707] Key type lgssc registered [ 1341.355785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.797180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.009457] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.042661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.322325] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1352.160486] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.791605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.038806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.726715] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1362.672566] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1366.247298] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1370.480717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.682709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.558888] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1376.811086] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:385 [ 1378.392231] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1386.314228] Lustre: Failing over lustre-MDT0000 [ 1386.447975] Lustre: server umount lustre-MDT0000 complete [ 1390.050779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.056862] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.061865] Lustre: Skipped 2 previous similar messages [ 1390.078440] LustreError: Skipped 1 previous similar message [ 1391.508846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.583678] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.584032] LustreError: 11-0: MGC192.168.201.121@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1391.590677] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0xae42999210f2af56 to 0xae42999210f2b338 [ 1391.602695] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 0@lo) [ 1391.662138] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1391.713695] LustreError: 79266:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-9) [ 1391.715735] LustreError: 79266:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -9 [ 1391.719762] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1391.719762] [ 1391.724837] LustreError: 15c-8: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1391.731872] LustreError: 79254:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1391.737403] LustreError: 79254:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -9 [ 1391.741839] LustreError: 79254:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 1391.762508] Lustre: server umount lustre-MDT0000 complete [ 1391.765303] LustreError: 79254:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1395.457948] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.469757] LustreError: Skipped 5 previous similar messages [ 1395.902940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.977602] LustreError: 11-0: MGC192.168.201.121@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1395.981670] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.989877] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0xae42999210f2b338 to 0xae42999210f2b3d2 [ 1395.999492] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 0@lo) [ 1396.134289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.174267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1397.475915] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1398.608608] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1401.319244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1401.329660] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1401.353299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:417 [ 1405.411622] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1405.419903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.441664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1406.434082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.435429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1406.448163] Lustre: Skipped 1 previous similar message [ 1409.693478] Lustre: server umount lustre-OST0000 complete [ 1416.673267] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1416.674757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.675426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.682362] LustreError: Skipped 1 previous similar message [ 1416.695251] Lustre: Skipped 1 previous similar message [ 1421.794544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.801043] Lustre: Skipped 2 previous similar messages [ 1426.399598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 1426.475954] Lustre: server umount lustre-MDT0000 complete [ 1426.913359] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.919255] LustreError: Skipped 4 previous similar messages [ 1428.907617] LustreError: 76089:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323192 with bad export cookie 12556767563690980306 [ 1428.911620] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1436.660323] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1438.356370] Key type lgssc unregistered [ 1438.563985] LNet: 81150:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.569926] LNet: Removed LNI 192.168.201.121@tcp [ 1439.078781] Key type .llcrypt unregistered [ 1439.080788] Key type ._llcrypt unregistered [ 1452.159668] alg: No test for adler32 (adler32-zlib) [ 1452.912556] Key type ._llcrypt registered [ 1452.913918] Key type .llcrypt registered [ 1452.963415] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1453.437809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1453.662106] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1453.780404] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1453.783338] LNet: Accept secure, port 988 [ 1455.391245] Key type lgssc registered [ 1455.932926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1460.290700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.560259] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.587397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.876557] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1466.854334] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.232543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.442590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.951385] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1476.399863] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1480.027652] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1483.925180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1484.104299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.210092] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1490.275741] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1492.331111] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:449 [ 1500.642568] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.646406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.653501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1502.690154] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1502.690198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1502.695096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.703563] Lustre: Skipped 1 previous similar message [ 1505.789576] Lustre: server umount lustre-OST0000 complete [ 1510.368550] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1510.371901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.379443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1512.929512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1512.929553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1514.050617] Lustre: server umount lustre-MDT0000 complete [ 1516.584084] LustreError: 82591:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323280 with bad export cookie 9463906295160470914 [ 1516.590958] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1516.592225] LustreError: 82591:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1516.830742] Lustre: server umount lustre-MDT0001 complete [ 1525.067743] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1526.553583] Key type lgssc unregistered [ 1526.720482] LNet: 86261:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.733176] LNet: Removed LNI 192.168.201.121@tcp [ 1527.185091] Key type .llcrypt unregistered [ 1527.186814] Key type ._llcrypt unregistered [ 1539.940867] alg: No test for adler32 (adler32-zlib) [ 1540.693479] Key type ._llcrypt registered [ 1540.694825] Key type .llcrypt registered [ 1540.758345] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1541.265051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.454898] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1541.566824] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1541.571884] LNet: Accept secure, port 988 [ 1543.175123] Key type lgssc registered [ 1543.689974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1548.295211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.545462] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1549.584510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.827284] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1554.918291] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1556.657128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.936983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1559.311778] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1563.509765] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1566.663231] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1570.182511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1570.370857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.646540] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1576.611640] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1579.627059] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:481 [ 1588.194544] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1588.201197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.209396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1590.240313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1590.243481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.249939] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1591.414837] Lustre: server umount lustre-OST0000 complete [ 1595.362229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.368577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.370941] Lustre: Skipped 2 previous similar messages [ 1599.855404] Lustre: server umount lustre-MDT0000 complete [ 1600.481832] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.495545] LustreError: Skipped 1 previous similar message [ 1602.299855] LustreError: 90572:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323365 with bad export cookie 103078751587933602 [ 1602.300970] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1602.307304] LustreError: 90572:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1602.539826] Lustre: server umount lustre-MDT0001 complete [ 1609.900924] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1611.466124] Key type lgssc unregistered [ 1611.647631] LNet: 91374:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.653475] LNet: Removed LNI 192.168.201.121@tcp [ 1612.097809] Key type .llcrypt unregistered [ 1612.099788] Key type ._llcrypt unregistered [ 1625.054365] alg: No test for adler32 (adler32-zlib) [ 1625.806486] Key type ._llcrypt registered [ 1625.809759] Key type .llcrypt registered [ 1625.880736] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1626.365556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1626.625783] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1626.770959] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1626.777552] LNet: Accept secure, port 988 [ 1628.383381] Key type lgssc registered [ 1628.998409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1633.305974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.535687] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.567402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.732605] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1639.906310] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.056839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.264360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1643.428633] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1647.478408] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1650.847532] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1655.049992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1655.252064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.833193] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1661.912424] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1664.496100] Lustre: lustre-OST0000: deleting orphan objects from 0x0:451 to 0x0:513 [ 1673.188270] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1673.193861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.205726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.722265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.727623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.392288] Lustre: server umount lustre-OST0000 complete [ 1682.399988] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1682.404055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.414909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.418911] Lustre: Skipped 1 previous similar message [ 1684.961451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1684.962833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.874913] Lustre: server umount lustre-MDT0000 complete [ 1688.868742] LustreError: 95484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323452 with bad export cookie 367358211716522045 [ 1688.872013] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.874606] LustreError: 95484:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1689.046445] Lustre: server umount lustre-MDT0001 complete [ 1696.074153] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1697.597473] Key type lgssc unregistered [ 1697.802693] LNet: 96492:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.806775] LNet: Removed LNI 192.168.201.121@tcp [ 1698.275138] Key type .llcrypt unregistered [ 1698.276775] Key type ._llcrypt unregistered [ 1711.712042] alg: No test for adler32 (adler32-zlib) [ 1712.464523] Key type ._llcrypt registered [ 1712.465837] Key type .llcrypt registered [ 1712.526140] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1713.097934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.318280] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1713.442890] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1713.447788] LNet: Accept secure, port 988 [ 1715.063130] Key type lgssc registered [ 1715.527217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.108571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.671416] LustreError: 97931:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 1767.391719] LustreError: 97931:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 1797.087535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1799.200954] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1803.268387] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1806.071212] Lustre: server umount lustre-OST0000 complete [ 1810.032393] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:32:52 (1761323572) [ 1812.974924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.177324] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.201077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.044835] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1818.592609] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.315082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.521430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.485825] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1825.251965] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1828.339069] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1831.543890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1831.751277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.043958] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1837.838723] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1844.080875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:545 [ 1854.433987] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.434175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.437932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1854.446378] Lustre: Skipped 1 previous similar message [ 1854.451444] Lustre: Skipped 1 previous similar message [ 1859.554248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.557223] Lustre: Skipped 1 previous similar message [ 1859.767978] Lustre: server umount lustre-OST0000 complete [ 1864.672250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.673390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.678669] Lustre: Skipped 1 previous similar message [ 1867.810049] Lustre: server umount lustre-MDT0000 complete [ 1869.792217] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.805083] LustreError: Skipped 1 previous similar message [ 1869.877657] LustreError: 101477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323633 with bad export cookie 6241766007829090534 [ 1869.880259] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.888601] LustreError: 101477:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1870.095118] Lustre: server umount lustre-MDT0001 complete [ 1876.681629] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1877.898209] Key type lgssc unregistered [ 1878.056455] LNet: 103327:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.060438] LNet: Removed LNI 192.168.201.121@tcp [ 1878.463787] Key type .llcrypt unregistered [ 1878.467405] Key type ._llcrypt unregistered [ 1891.087059] alg: No test for adler32 (adler32-zlib) [ 1891.838576] Key type ._llcrypt registered [ 1891.840527] Key type .llcrypt registered [ 1891.893810] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1892.386783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1892.551712] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1892.648233] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1892.651794] LNet: Accept secure, port 988 [ 1894.239156] Key type lgssc registered [ 1894.768426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1898.601216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.831303] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.863975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.811592] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1905.121748] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.454769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.651311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1908.929249] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1913.042681] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1915.902702] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1919.098411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1919.261647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.362382] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 1924.955783] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1929.511555] Lustre: lustre-OST0000: deleting orphan objects from 0x0:516 to 0x0:577 [ 1934.818570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.819293] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1934.827161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.832187] Lustre: Skipped 1 previous similar message [ 1936.336312] Lustre: server umount lustre-OST0000 complete [ 1939.936766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.940520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.943256] Lustre: Skipped 2 previous similar messages [ 1944.537287] Lustre: server umount lustre-MDT0000 complete [ 1945.057636] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.064614] LustreError: Skipped 1 previous similar message [ 1946.544085] LustreError: 104766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323710 with bad export cookie 5799855170505829799 [ 1946.548085] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.552617] LustreError: 104766:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1946.713515] Lustre: server umount lustre-MDT0001 complete [ 1952.824893] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1954.050501] Key type lgssc unregistered [ 1954.193782] LNet: 108439:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.198943] LNet: Removed LNI 192.168.201.121@tcp [ 1954.565790] Key type .llcrypt unregistered [ 1954.567263] Key type ._llcrypt unregistered [ 1963.835440] alg: No test for adler32 (adler32-zlib) [ 1964.588158] Key type ._llcrypt registered [ 1964.589864] Key type .llcrypt registered [ 1964.666766] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1965.133374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1965.474140] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 1965.620430] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1965.624781] LNet: Accept secure, port 988 [ 1967.239864] Key type lgssc registered [ 1967.874166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.480194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.023308] LustreError: 109729:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 2020.767221] LustreError: 109729:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2050.463179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.197263] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2055.461590] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2058.283588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.070761] LustreError: 15c-8: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2073.077332] LustreError: 110714:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2073.081528] LustreError: 110714:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 2073.086633] LustreError: 110714:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 2073.159435] Lustre: server umount lustre-MDT0000 complete [ 2073.161610] LustreError: 110714:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2082.783161] Lustre: 111147:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761323840/real 1761323840] req@000000005d5ec795 x1846881790461888/t0(0) o251->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761323846 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2082.860889] Lustre: server umount lustre-OST0000 complete [ 2087.259804] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2091.188876] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2095.672199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2098.728645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2101.611430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.500060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2107.831385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2107.865084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.999352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2108.018810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2108.068928] Lustre: lustre-MDT0000: new disk, initializing [ 2108.126517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.136571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2110.049196] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2115.336480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2115.371293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.417425] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2115.441037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2115.443852] Lustre: Skipped 1 previous similar message [ 2115.487623] Lustre: lustre-MDT0001: new disk, initializing [ 2115.529030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.548075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2115.552794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2117.523198] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2122.182870] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2124.805839] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2127.958623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2128.002758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.005787] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2128.129549] Lustre: lustre-OST0000: new disk, initializing [ 2128.132875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2128.173790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.141194] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2134.521461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2134.525849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2134.875153] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2136.757546] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2136.856957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2138.674784] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2138.767797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2144.736288] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.740657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.746183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.748241] Lustre: Skipped 1 previous similar message [ 2145.767551] Lustre: server umount lustre-OST0000 complete [ 2149.857192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.858240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.860937] Lustre: Skipped 2 previous similar messages [ 2153.360502] Lustre: server umount lustre-MDT0000 complete [ 2154.973195] LustreError: 113590:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761323918 with bad export cookie 8738987275051980022 [ 2154.976552] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.977974] LustreError: 113590:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2158.703993] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 12:38:41 (1761323921) [ 2160.077383] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2166.846949] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2170.432449] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2174.991445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2178.504966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2181.257857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2184.038212] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.090289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.139502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.254088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2187.277511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2187.319830] Lustre: lustre-MDT0000: new disk, initializing [ 2187.364790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.374063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2189.145532] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2194.444798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2194.496604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.563366] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2194.596040] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2194.598867] Lustre: Skipped 1 previous similar message [ 2194.642575] Lustre: lustre-MDT0001: new disk, initializing [ 2194.703240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2194.707573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2196.588989] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2201.468145] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2204.013901] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2206.982187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2207.020289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2207.023538] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2207.138365] Lustre: lustre-OST0000: new disk, initializing [ 2207.140419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2207.181970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.185534] Lustre: Skipped 1 previous similar message [ 2208.997851] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2209.919738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2209.923543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2213.563498] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2215.159702] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2215.246640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2216.962845] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2217.062852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2220.515308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.515465] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2220.516027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.516033] Lustre: Skipped 1 previous similar message [ 2220.520254] Lustre: Skipped 1 previous similar message [ 2220.523991] LustreError: Skipped 1 previous similar message [ 2225.634407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2225.637191] Lustre: Skipped 2 previous similar messages [ 2230.754096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.756812] Lustre: Skipped 1 previous similar message [ 2232.287114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2232.375113] Lustre: server umount lustre-OST0000 complete [ 2232.378879] Lustre: Skipped 1 previous similar message [ 2235.874252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.874478] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2235.880496] Lustre: Skipped 1 previous similar message [ 2235.883965] LustreError: Skipped 1 previous similar message [ 2240.532496] Lustre: server umount lustre-MDT0000 complete [ 2240.993331] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.997795] LustreError: Skipped 1 previous similar message [ 2242.215585] LustreError: 120713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324005 with bad export cookie 8738987275051980862 [ 2242.217044] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.221978] LustreError: 120713:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2246.194576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.392409] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.425929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.293202] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2251.745200] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.166426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.196384] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2257.402445] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2259.874818] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2262.808967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.696561] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2267.661763] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2281.054205] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2284.002117] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2284.005846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.011840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.014328] Lustre: Skipped 2 previous similar messages [ 2288.621835] Lustre: server umount lustre-OST0000 complete [ 2288.623744] Lustre: Skipped 1 previous similar message [ 2292.704282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.709401] Lustre: Skipped 1 previous similar message [ 2297.824683] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.835791] LustreError: Skipped 1 previous similar message [ 2298.216046] LustreError: 125867:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324061 with bad export cookie 8738987275051981709 [ 2298.219886] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2298.220770] LustreError: 125867:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2303.980950] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2305.250050] Key type lgssc unregistered [ 2305.396622] LNet: 128850:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.401761] LNet: Removed LNI 192.168.201.121@tcp [ 2305.757232] Key type .llcrypt unregistered [ 2305.758715] Key type ._llcrypt unregistered [ 2314.097611] alg: No test for adler32 (adler32-zlib) [ 2314.849360] Key type ._llcrypt registered [ 2314.850515] Key type .llcrypt registered [ 2314.897349] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2321.733856] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2322.104301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.319124] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2322.407265] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2322.409809] LNet: Accept secure, port 988 [ 2323.999126] Key type lgssc registered [ 2324.403862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.721682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2330.226538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2332.508981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2334.826645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2337.911290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2337.954957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.078036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2339.094690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 2339.136499] Lustre: lustre-MDT0000: new disk, initializing [ 2339.184542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.196591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2341.040791] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2345.895403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2345.921457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.967952] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2345.989108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 2345.991531] Lustre: Skipped 1 previous similar message [ 2346.031564] Lustre: lustre-MDT0001: new disk, initializing [ 2346.072784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.088500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2346.092647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2347.912938] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2352.368090] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2354.809463] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2357.417456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2357.445101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.448828] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 2357.544205] Lustre: lustre-OST0000: new disk, initializing [ 2357.546817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 2357.580269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2359.215504] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2363.498848] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2364.924487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2364.928323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2365.186876] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2365.278831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2366.934354] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2367.034100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2370.016593] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.017333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.021169] LustreError: Skipped 1 previous similar message [ 2370.028721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.032152] Lustre: Skipped 1 previous similar message [ 2373.939420] Lustre: server umount lustre-OST0000 complete [ 2376.671816] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2376.675683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.681371] Lustre: Skipped 1 previous similar message [ 2376.684146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.256324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.256687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.513107] Lustre: server umount lustre-MDT0000 complete [ 2383.157278] LustreError: 132114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324146 with bad export cookie 8757832319428111129 [ 2383.159117] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.161601] LustreError: 132114:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.310143] Lustre: server umount lustre-MDT0001 complete [ 2386.744306] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 12:42:29 (1761324149) [ 2389.300116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.444785] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.465130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.115809] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2394.147548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.294067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2396.036600] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2399.295536] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2401.793771] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2404.832748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.832812] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2404.837924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.844397] Lustre: Skipped 2 previous similar messages [ 2408.773553] Lustre: server umount lustre-MDT0000 complete [ 2409.952493] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.958201] LustreError: Skipped 1 previous similar message [ 2410.316326] LustreError: 138348:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324173 with bad export cookie 8757832319428111976 [ 2410.319534] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.319682] LustreError: 138348:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2413.626264] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 12:42:56 (1761324176) [ 2416.217406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2433.695572] LustreError: 139193:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 2444.959590] LustreError: 139193:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2473.631717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.275046] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2478.794774] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2479.809397] Lustre: server umount lustre-OST0000 complete [ 2479.811522] Lustre: Skipped 1 previous similar message [ 2483.047317] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 12:44:05 (1761324245) [ 2485.425823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.582987] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.156534] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2490.361212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.499330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2490.502602] Lustre: Skipped 1 previous similar message [ 2492.134741] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2495.035741] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.127877] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2499.495361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2501.051329] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2503.762243] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2515.936669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.938256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2515.941772] Lustre: Skipped 3 previous similar messages [ 2515.944023] Lustre: Skipped 1 previous similar message [ 2517.581553] Lustre: server umount lustre-MDT0000 complete [ 2519.110381] LustreError: 140575:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324282 with bad export cookie 8757832319428112627 [ 2519.112814] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.117990] LustreError: 140575:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2527.199143] Lustre: 143643:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324284/real 1761324284] req@00000000478c7e97 x1846882163786816/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324290 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2530.512204] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 12:44:53 (1761324293) [ 2533.190930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.406843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.435614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.438385] Lustre: Skipped 1 previous similar message [ 2535.046416] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2538.018351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.520915] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2542.098594] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.083311] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2546.245405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.876333] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2549.413745] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:65 [ 2550.571622] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2552.165702] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2552.255419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2553.754184] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2553.834692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2554.847881] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2554.851707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.856048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.858255] Lustre: Skipped 1 previous similar message [ 2560.617853] Lustre: server umount lustre-OST0000 complete [ 2560.619796] Lustre: Skipped 2 previous similar messages [ 2573.280450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.283444] Lustre: Skipped 7 previous similar messages [ 2576.351138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2577.726907] LustreError: 144294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324341 with bad export cookie 8757832319428113677 [ 2577.729349] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.731328] LustreError: 144294:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2580.763918] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 12:45:43 (1761324343) [ 2582.954954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.415600] LustreError: 148640:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 2611.679496] LustreError: 148640:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2640.352652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.356050] Lustre: Skipped 2 previous similar messages [ 2641.867738] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2644.446091] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2646.781253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.798083] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.841502] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:97 [ 2659.284556] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2662.314262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.871707] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2666.457137] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2668.348790] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2669.762369] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2669.831814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2671.179756] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2671.250311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2672.610985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.611099] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2672.611588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.611594] Lustre: Skipped 2 previous similar messages [ 2672.613993] Lustre: Skipped 5 previous similar messages [ 2678.024352] Lustre: server umount lustre-MDT0000 complete [ 2678.025818] Lustre: Skipped 2 previous similar messages [ 2679.532529] LustreError: 151951:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324443 with bad export cookie 8757832319428114692 [ 2679.535246] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2679.537472] LustreError: 151951:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2687.455167] Lustre: 152353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324444/real 1761324444] req@00000000ae697c2b x1846882163808704/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761324450 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2690.577389] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 12:47:33 (1761324453) [ 2693.118149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.687639] LustreError: 152996:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 2721.951421] LustreError: 152996:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2752.001242] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2754.578220] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2756.937840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.480493] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.486690] LustreError: Skipped 2 previous similar messages [ 2769.506577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.508901] Lustre: Skipped 3 previous similar messages [ 2769.532144] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:129 [ 2770.997241] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2773.833915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.377622] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2777.786675] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2779.672922] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2781.918261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.946458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.949730] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 2782.008392] Lustre: lustre-OST0001: new disk, initializing [ 2782.010148] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 2783.415393] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2787.324235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2787.330045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2787.429556] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2788.962160] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 2789.035232] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2790.447438] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 2790.530660] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2794.466242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.466966] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2794.469877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2794.473192] Lustre: Skipped 6 previous similar messages [ 2797.303448] Lustre: server umount lustre-OST0000 complete [ 2797.305306] Lustre: Skipped 2 previous similar messages [ 2799.585319] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.589279] LustreError: Skipped 1 previous similar message [ 2813.343983] LustreError: 153004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324576 with bad export cookie 8757832319428115630 [ 2813.347098] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2813.348735] LustreError: 153004:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2815.486958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2816.862093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2818.020075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2819.256256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2821.528844] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 12:49:44 (1761324584) [ 2822.157373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2822.765529] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2823.364790] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 12:49:45 (1761324585) [ 2825.594296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.688110] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2825.699295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2827.072108] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2829.802570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.835444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2829.846684] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2831.309111] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2833.834668] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2835.664457] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2838.028124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2838.104391] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2839.622287] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2842.062354] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2843.642836] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2844.166462] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:161 [ 2844.760440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2846.261869] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2846.354335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2849.250227] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.252424] LustreError: Skipped 2 previous similar messages [ 2866.934829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.518510] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2870.982984] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2872.629189] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 2881.379210] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.382590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2881.382677] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.121@tcp (at 0@lo) [ 2881.384564] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:162 to 0x280000400:193 [ 2881.391783] Lustre: lustre-OST0000: deleting orphan objects from 0x0:36 to 0x0:193 [ 2881.394334] Lustre: Skipped 1 previous similar message [ 2881.889868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2883.141353] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 2883.202329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2885.110399] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c52479dd000.ost_server_uuid 40 [ 2885.656384] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c52479dd000.ost_server_uuid in FULL state after 0 sec [ 2885.718854] LustreError: 159575:0:(ldlm_lockd.c:721:ldlm_handle_ast_error()) ### client (nid 192.168.201.21@tcp) returned error from blocking AST (req@00000000a3236fda x1846882163846464 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000fa014581/0x798a1217f1f2c5b4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.21@tcp remote: 0xd222cc92b0b6041 expref: 6 pid: 161547 timeout: 2985 lvb_type: 0 [ 2885.734941] LustreError: 138-a: lustre-MDT0000: A client on nid 192.168.201.21@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2885.739813] LustreError: 159565:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.21@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000fa014581/0x798a1217f1f2c5b4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.21@tcp remote: 0xd222cc92b0b6041 expref: 7 pid: 161547 timeout: 0 lvb_type: 0 [ 2891.744918] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.746070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.747703] LustreError: Skipped 2 previous similar messages [ 2891.751390] Lustre: Skipped 8 previous similar messages [ 2900.959131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2909.685137] LustreError: 159562:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324673 with bad export cookie 8757832319428116771 [ 2909.687722] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.688040] LustreError: 159562:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2920.604972] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2921.629670] Key type lgssc unregistered [ 2921.746395] LNet: 166027:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.750077] LNet: Removed LNI 192.168.201.121@tcp [ 2922.046491] Key type .llcrypt unregistered [ 2922.048837] Key type ._llcrypt unregistered [ 2931.060388] alg: No test for adler32 (adler32-zlib) [ 2931.812471] Key type ._llcrypt registered [ 2931.814470] Key type .llcrypt registered [ 2931.862217] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2932.292870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2932.473670] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 2932.560438] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2932.562117] LNet: Accept secure, port 988 [ 2934.143116] Key type lgssc registered [ 2934.516811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2937.210988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.363848] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.379627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.905138] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2942.814536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.942472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.393621] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2946.804178] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.520614] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2950.688462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2950.791952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.220802] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2953.893359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 2954.698805] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2961.809949] Lustre: Failing over lustre-MDT0000 [ 2961.932313] Lustre: server umount lustre-MDT0000 complete [ 2963.425024] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2963.425466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.426158] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.437262] Lustre: Skipped 2 previous similar messages [ 2968.544146] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.548066] LustreError: Skipped 4 previous similar messages [ 2969.567154] Lustre: 166800:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324726/real 1761324726] req@000000005092a726 x1846882804442304/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324733 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 2969.574082] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.930591] LustreError: 167466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.21@tcp arrived at 1761324736 with bad export cookie 13788656641664621360 [ 2972.936334] LustreError: 167466:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2974.687989] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.692672] LustreError: Skipped 5 previous similar messages [ 2975.457935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.713775] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0xbf5b2629004017e1 to 0xbf5b262900401bb5 [ 2975.718038] LustreError: 170612:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.201.121@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000006a92756f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2975.722479] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 0@lo) [ 2975.800559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.829789] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2977.228180] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 2977.963963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2977.967134] Lustre: lustre-MDT0000: Denying connection for new client 81acd5fe-9df8-4f20-a3ea-d0a56d72859e (at 192.168.201.21@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2980.836741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2985.952870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.954256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2985.958908] Lustre: Skipped 1 previous similar message [ 2991.072934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2991.075886] Lustre: Skipped 2 previous similar messages [ 2996.192138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2996.194461] Lustre: Skipped 1 previous similar message [ 2999.263176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 2999.327750] Lustre: server umount lustre-OST0000 complete [ 3000.654804] LustreError: 171336:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3000.659153] Lustre: 170633:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3000.663535] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 3000.680155] Lustre: 170633:0:(mdt_handler.c:7532:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 3001.312456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3001.316867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.432118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.786833] Lustre: server umount lustre-MDT0000 complete [ 3008.004931] LustreError: 167467:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324771 with bad export cookie 13788656641664621493 [ 3008.007367] LustreError: 166797:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3008.007522] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.009810] LustreError: 167467:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3022.303083] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3022.410303] Lustre: server umount lustre-MDT0001 complete [ 3027.054332] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3028.138193] Key type lgssc unregistered [ 3028.265395] LNet: 172141:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3028.268844] LNet: Removed LNI 192.168.201.121@tcp [ 3028.607346] Key type .llcrypt unregistered [ 3028.608420] Key type ._llcrypt unregistered [ 3037.835927] alg: No test for adler32 (adler32-zlib) [ 3038.587373] Key type ._llcrypt registered [ 3038.588319] Key type .llcrypt registered [ 3038.627617] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3038.984958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3039.235736] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3039.323101] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3039.324887] LNet: Accept secure, port 988 [ 3040.919174] Key type lgssc registered [ 3041.335369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3044.213914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.390749] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.405652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.008694] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3049.834907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.950327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.245328] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3053.665097] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3055.501535] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3057.699729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3057.802092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.262663] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3060.899994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:257 [ 3061.509811] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3064.995134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.995510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.724992] Lustre: server umount lustre-OST0000 complete [ 3075.552564] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3075.552601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.553176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.553180] Lustre: Skipped 1 previous similar message [ 3075.562720] Lustre: Skipped 1 previous similar message [ 3077.192510] Lustre: server umount lustre-MDT0000 complete [ 3078.405987] LustreError: 173580:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324841 with bad export cookie 8217230157922210329 [ 3078.408339] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3078.409083] LustreError: 173580:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3078.535765] Lustre: server umount lustre-MDT0001 complete [ 3083.119117] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3084.207247] Key type lgssc unregistered [ 3084.314456] LNet: 177251:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3084.317656] LNet: Removed LNI 192.168.201.121@tcp [ 3084.650826] Key type .llcrypt unregistered [ 3084.652780] Key type ._llcrypt unregistered [ 3096.987618] alg: No test for adler32 (adler32-zlib) [ 3097.739354] Key type ._llcrypt registered [ 3097.741337] Key type .llcrypt registered [ 3097.790819] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3098.138304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3098.385362] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3098.472768] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3098.474952] LNet: Accept secure, port 988 [ 3100.063097] Key type lgssc registered [ 3100.438990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.359393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.533817] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.551760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.069370] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3108.873536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.983806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3110.393321] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3112.875059] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3114.673711] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3116.861633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3116.957701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.264469] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3120.039321] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:289 [ 3120.644962] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3129.312536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.314041] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3129.314990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3129.316943] Lustre: Skipped 1 previous similar message [ 3134.122097] Lustre: server umount lustre-OST0000 complete [ 3139.552313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.553096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.558274] Lustre: Skipped 1 previous similar message [ 3139.561137] Lustre: Skipped 1 previous similar message [ 3141.512054] Lustre: server umount lustre-MDT0000 complete [ 3142.865100] LustreError: 180949:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324906 with bad export cookie 7126328511570307160 [ 3142.868018] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.868726] LustreError: 180949:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3143.027047] Lustre: server umount lustre-MDT0001 complete [ 3147.722882] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3148.770336] Key type lgssc unregistered [ 3148.884328] LNet: 182651:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.886600] LNet: Removed LNI 192.168.201.121@tcp [ 3149.175971] Key type .llcrypt unregistered [ 3149.176926] Key type ._llcrypt unregistered [ 3157.460081] alg: No test for adler32 (adler32-zlib) [ 3158.212355] Key type ._llcrypt registered [ 3158.213301] Key type .llcrypt registered [ 3158.250361] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3158.620677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3158.843774] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3158.924742] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3158.926914] LNet: Accept secure, port 988 [ 3160.503143] Key type lgssc registered [ 3160.873142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3164.644961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.786633] Lustre: *** cfs_fail_loc=135, val=0*** [ 3165.826248] LustreError: 184220:0:(obd_config.c:774:class_setup()) setup lustre-MDT0000 failed (-2) [ 3165.829279] LustreError: 184220:0:(obd_config.c:1999:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -2 [ 3165.832337] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3165.832337] [ 3165.835370] LustreError: 15c-8: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3165.842423] LustreError: 184182:0:(obd_mount_server.c:1425:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3165.845301] LustreError: 184182:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [ 3165.848402] LustreError: 184182:0:(obd_config.c:827:class_cleanup()) Device 5 not setup [ 3172.319144] Lustre: 184182:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761324929/real 1761324929] req@00000000b31027bb x1846883041411904/t0(0) o251->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761324935 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3172.373824] Lustre: server umount lustre-MDT0000 complete [ 3172.374996] LustreError: 184182:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3174.477299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.610380] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.624466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.904539] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3178.600444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.732547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3180.091991] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3182.568156] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3184.400846] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3189.216282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.216349] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3189.216963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.219464] Lustre: Skipped 1 previous similar message [ 3191.163664] Lustre: server umount lustre-MDT0000 complete [ 3192.324913] LustreError: 184572:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761324955 with bad export cookie 7878066122450012029 [ 3192.327314] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.327578] LustreError: 184572:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3192.437173] Lustre: server umount lustre-MDT0001 complete [ 3195.659118] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3196.543847] Key type lgssc unregistered [ 3196.654254] LNet: 187074:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3196.656933] LNet: Removed LNI 192.168.201.121@tcp [ 3196.925578] Key type .llcrypt unregistered [ 3196.926706] Key type ._llcrypt unregistered [ 3204.570085] alg: No test for adler32 (adler32-zlib) [ 3205.321598] Key type ._llcrypt registered [ 3205.323214] Key type .llcrypt registered [ 3205.364992] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3205.468626] Key type .llcrypt unregistered [ 3205.469626] Key type ._llcrypt unregistered [ 3211.786589] alg: No test for adler32 (adler32-zlib) [ 3212.538428] Key type ._llcrypt registered [ 3212.539401] Key type .llcrypt registered [ 3212.576837] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3212.913642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3213.092368] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3213.163471] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3213.165414] LNet: Accept secure, port 988 [ 3214.743141] Key type lgssc registered [ 3215.073855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3217.587585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3234.079286] LustreError: 189325:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 1: rc = -5 [ 3265.823231] LustreError: 189325:0:(mgc_request.c:255:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 3295.519177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.943183] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3299.361328] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3301.488820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.176976] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.199852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.217528] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:321 [ 3320.570710] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3323.274460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.386052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3324.780183] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3327.242163] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3329.009382] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3329.954491] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3338.721605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.721838] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3338.722726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.724502] Lustre: Skipped 1 previous similar message [ 3343.841111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3343.843478] Lustre: Skipped 2 previous similar messages [ 3348.959137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3348.962328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3348.965485] Lustre: Skipped 1 previous similar message [ 3349.031559] Lustre: server umount lustre-OST0000 complete [ 3354.081323] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3354.081328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.082179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.091183] Lustre: Skipped 1 previous similar message [ 3356.482476] Lustre: server umount lustre-MDT0000 complete [ 3357.802235] LustreError: 189333:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325121 with bad export cookie 6876578937379078782 [ 3357.805277] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3357.805737] LustreError: 189333:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3357.936994] Lustre: server umount lustre-MDT0001 complete [ 3361.502713] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3362.585918] Key type lgssc unregistered [ 3362.701368] LNet: 193366:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3362.705500] LNet: Removed LNI 192.168.201.121@tcp [ 3362.977888] Key type .llcrypt unregistered [ 3362.978874] Key type ._llcrypt unregistered [ 3371.767313] alg: No test for adler32 (adler32-zlib) [ 3372.518455] Key type ._llcrypt registered [ 3372.520106] Key type .llcrypt registered [ 3372.561384] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3373.044562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3373.189014] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3373.272079] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3373.274388] LNet: Accept secure, port 988 [ 3374.855166] Key type lgssc registered [ 3375.270181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3378.059754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.221843] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3379.238591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.711605] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3383.644081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3383.778274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3385.326532] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3388.019135] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3389.935402] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3392.295164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3392.414526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3393.787277] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3394.471983] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:353 [ 3396.125780] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3398.273467] Lustre: Failing over lustre-MDT0000 [ 3398.349952] Lustre: server umount lustre-MDT0000 complete [ 3398.561477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.567303] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.575822] LustreError: Skipped 2 previous similar messages [ 3402.500614] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.612435] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.616232] LustreError: Skipped 3 previous similar messages [ 3410.911083] Lustre: 194140:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761325167/real 1761325167] req@00000000d95aea4a x1846883265815360/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761325174 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'' [ 3410.916642] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.626484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.780264] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.786881] LustreError: Skipped 7 previous similar messages [ 3417.057165] Lustre: Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0xcc486cd05658cb10 to 0xcc486cd05658cedd [ 3417.063106] LustreError: 198000:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.201.121@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000f1ebdfb2) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3417.072379] Lustre: MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 0@lo) [ 3417.143421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.170089] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3418.526804] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3420.044056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 3420.924253] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3422.180314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3422.181130] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3422.198885] Lustre: lustre-OST0000: deleting orphan objects from 0x0:291 to 0x0:385 [ 3423.266533] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 3432.418778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.423840] Lustre: Skipped 2 previous similar messages [ 3432.425463] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3432.425706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3432.432054] Lustre: Skipped 1 previous similar message [ 3436.769626] Lustre: server umount lustre-OST0000 complete [ 3442.657070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.657578] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3442.657872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.660912] Lustre: Skipped 2 previous similar messages [ 3447.775839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.777756] Lustre: Skipped 2 previous similar messages [ 3452.383185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3452.428538] Lustre: server umount lustre-MDT0000 complete [ 3453.760459] LustreError: 198563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325217 with bad export cookie 14720135024118124253 [ 3453.763600] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3453.765548] LustreError: 198563:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3458.566926] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3459.496751] Key type lgssc unregistered [ 3459.616548] LNet: 199876:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3459.620313] LNet: Removed LNI 192.168.201.121@tcp [ 3459.911020] Key type .llcrypt unregistered [ 3459.912236] Key type ._llcrypt unregistered [ 3468.452212] alg: No test for adler32 (adler32-zlib) [ 3469.203384] Key type ._llcrypt registered [ 3469.204494] Key type .llcrypt registered [ 3469.247345] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3469.665129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3469.862970] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3469.941887] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3469.943493] LNet: Accept secure, port 988 [ 3471.519115] Key type lgssc registered [ 3471.875229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3474.687427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.850574] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.867334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.399837] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3480.244090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.350138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3481.810664] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3484.235777] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3485.969188] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3488.068247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3488.165477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.465122] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3491.239196] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:417 [ 3491.777943] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3493.316365] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3500.836072] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3509.879174] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3522.015858] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3522.018717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.023764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.113468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3526.113582] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3526.116588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.458062] Lustre: server umount lustre-OST0000 complete [ 3531.232572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.233044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.236947] Lustre: Skipped 1 previous similar message [ 3531.239051] Lustre: Skipped 1 previous similar message [ 3536.352027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3536.354398] Lustre: Skipped 2 previous similar messages [ 3541.471876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.475125] Lustre: Skipped 1 previous similar message [ 3543.007147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3543.048019] Lustre: server umount lustre-MDT0000 complete [ 3544.402111] LustreError: 203931:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325307 with bad export cookie 10562670179899075577 [ 3544.405752] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.407560] LustreError: 203931:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3544.560784] Lustre: server umount lustre-MDT0001 complete [ 3549.590219] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3550.569552] Key type lgssc unregistered [ 3550.683420] LNet: 205196:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3550.685585] LNet: Removed LNI 192.168.201.121@tcp [ 3550.977911] Key type .llcrypt unregistered [ 3550.978872] Key type ._llcrypt unregistered [ 3560.002082] alg: No test for adler32 (adler32-zlib) [ 3560.754440] Key type ._llcrypt registered [ 3560.755531] Key type .llcrypt registered [ 3560.794479] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3561.225947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.393218] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3561.472419] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3561.474959] LNet: Accept secure, port 988 [ 3563.063109] Key type lgssc registered [ 3563.402382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3566.046540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.206502] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.223079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.732193] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3571.792081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.932205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3573.422962] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3575.999080] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3577.947479] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3580.250801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3580.382290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.888234] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3583.461624] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:449 [ 3584.523814] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3586.962115] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3594.847096] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 3601.684477] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 3608.185387] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 3617.249366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.249937] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3617.250323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.252822] Lustre: Skipped 1 previous similar message [ 3622.369621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3622.372029] Lustre: Skipped 2 previous similar messages [ 3627.488815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3627.491191] Lustre: Skipped 1 previous similar message [ 3629.023155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 3629.090944] Lustre: server umount lustre-OST0000 complete [ 3632.608399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.609139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.611427] Lustre: Skipped 1 previous similar message [ 3636.551844] Lustre: server umount lustre-MDT0000 complete [ 3637.728105] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.734588] LustreError: Skipped 1 previous similar message [ 3637.914322] LustreError: 207443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325401 with bad export cookie 16768739488562217970 [ 3637.917088] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3637.919049] LustreError: 207443:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3638.047167] Lustre: server umount lustre-MDT0001 complete [ 3642.952510] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3644.089787] Key type lgssc unregistered [ 3644.209399] LNet: 212034:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3644.213860] LNet: Removed LNI 192.168.201.121@tcp [ 3644.570025] Key type .llcrypt unregistered [ 3644.571110] Key type ._llcrypt unregistered [ 3653.602806] alg: No test for adler32 (adler32-zlib) [ 3654.354427] Key type ._llcrypt registered [ 3654.355970] Key type .llcrypt registered [ 3654.390743] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3654.873259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3655.049564] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3655.128509] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3655.130487] LNet: Accept secure, port 988 [ 3656.719156] Key type lgssc registered [ 3657.066793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3659.648692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.801986] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.817242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.345936] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3665.397055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.529467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3667.021344] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3669.581232] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3671.547247] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3673.807804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.925508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.362543] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3677.029658] Lustre: lustre-OST0000: deleting orphan objects from 0x0:387 to 0x0:481 [ 3677.813777] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3681.211652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3681.236098] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3681.294021] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3682.793455] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3685.425245] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3689.016132] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3699.665216] Lustre: Permanently deactivating lustre-OST0001 [ 3699.672333] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3709.428596] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3717.668553] Lustre: Permanently reactivating lustre-OST0001 [ 3717.675249] Lustre: Modifying parameter lustre-OST0001-osc.osc.active in log lustre-client [ 3717.678577] Lustre: Skipped 2 previous similar messages [ 3725.807845] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.814321] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3725.817056] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3725.820349] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 3725.821119] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34 to 0x2c0000400:65 [ 3725.832452] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3725.838632] Lustre: lustre-OST0001: deleting orphan objects from 0x0:34 to 0x0:65 [ 3730.911973] LustreError: 11-0: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.912091] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.913933] LustreError: Skipped 1 previous similar message [ 3730.917992] Lustre: Skipped 1 previous similar message [ 3730.919644] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3730.921580] Lustre: Skipped 1 previous similar message [ 3732.961700] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3732.964956] Lustre: Skipped 1 previous similar message [ 3734.010159] Lustre: server umount lustre-OST0001 complete [ 3737.568068] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3737.571669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.577288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3738.082415] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.088827] LustreError: Skipped 1 previous similar message [ 3741.473177] Lustre: server umount lustre-OST0000 complete [ 3743.201339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.205133] Lustre: Skipped 1 previous similar message [ 3743.206905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3743.208606] Lustre: Skipped 3 previous similar messages [ 3748.320559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.322286] Lustre: Skipped 1 previous similar message [ 3748.936600] Lustre: server umount lustre-MDT0000 complete [ 3750.238042] LustreError: 216401:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325513 with bad export cookie 9971222968282956236 [ 3750.240385] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3750.243314] LustreError: 216401:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3755.166568] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3756.184472] Key type lgssc unregistered [ 3756.308339] LNet: 218636:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3756.311140] LNet: Removed LNI 192.168.201.121@tcp [ 3756.638665] Key type .llcrypt unregistered [ 3756.640366] Key type ._llcrypt unregistered [ 3768.372699] alg: No test for adler32 (adler32-zlib) [ 3769.124350] Key type ._llcrypt registered [ 3769.125454] Key type .llcrypt registered [ 3769.163913] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3775.055840] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3775.645552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3775.820839] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 3775.913392] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3775.915296] LNet: Accept secure, port 988 [ 3777.503158] Key type lgssc registered [ 3777.915503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3780.859704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3782.964897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3784.979773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3787.027312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3789.402414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3789.432147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.546519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3790.557995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 3790.600597] Lustre: lustre-MDT0000: new disk, initializing [ 3790.627762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.635758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3792.154140] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3796.425811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3796.450671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.482352] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3796.501871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 3796.504402] Lustre: Skipped 1 previous similar message [ 3796.545774] Lustre: lustre-MDT0001: new disk, initializing [ 3796.580472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3796.594122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3796.598591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3798.137752] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3801.969970] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3803.906047] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3806.240379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3806.264931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3806.268219] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 3806.374614] Lustre: lustre-OST0000: new disk, initializing [ 3806.377625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 3806.405693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.880660] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3811.737711] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3813.389964] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 3815.930457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3815.936498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3816.534841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3818.063698] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 3818.164583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3821.024282] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3821.027278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.033182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.036083] Lustre: Skipped 1 previous similar message [ 3825.002144] Lustre: server umount lustre-OST0000 complete [ 3827.167944] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3827.169847] LustreError: Skipped 1 previous similar message [ 3827.170926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.173734] Lustre: Skipped 1 previous similar message [ 3827.175076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.263961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.264359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.389135] Lustre: server umount lustre-MDT0000 complete [ 3833.665167] LustreError: 222726:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325597 with bad export cookie 5885920018602364392 [ 3833.667538] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.668449] LustreError: 222726:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3833.805902] Lustre: server umount lustre-MDT0001 complete [ 3836.483409] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 13:06:39 (1761325599) [ 3838.768956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.948771] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3838.973622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3840.439303] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3843.382318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.516863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.125183] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3847.766815] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3849.853878] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3852.224319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.357274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.892220] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 3856.578894] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3858.209972] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3863.769278] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3870.325673] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3884.414263] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3884.418218] Lustre: Skipped 1 previous similar message [ 3900.561800] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3900.566347] Lustre: Skipped 1 previous similar message [ 3932.851989] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 3932.854174] Lustre: Skipped 3 previous similar messages [ 4002.795218] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 4002.797888] Lustre: Skipped 7 previous similar messages [ 4024.799945] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4024.804308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.811557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4024.814564] Lustre: Skipped 1 previous similar message [ 4027.873126] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4030.622566] Lustre: server umount lustre-OST0000 complete [ 4032.992453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.992758] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4032.992981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.992985] Lustre: Skipped 2 previous similar messages [ 4032.995707] Lustre: Skipped 2 previous similar messages [ 4043.232428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4043.234599] Lustre: Skipped 4 previous similar messages [ 4046.303152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? [ 4046.353124] Lustre: server umount lustre-MDT0000 complete [ 4047.660265] LustreError: 227280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325811 with bad export cookie 5885920018602365232 [ 4047.663568] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4047.665479] LustreError: 227280:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4052.421293] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4053.378643] Key type lgssc unregistered [ 4053.484460] LNet: 232201:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4053.487627] LNet: Removed LNI 192.168.201.121@tcp [ 4053.785816] Key type .llcrypt unregistered [ 4053.787133] Key type ._llcrypt unregistered [ 4062.675095] alg: No test for adler32 (adler32-zlib) [ 4063.427321] Key type ._llcrypt registered [ 4063.428098] Key type .llcrypt registered [ 4063.469643] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4063.949677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4064.101128] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4064.168553] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4064.170214] LNet: Accept secure, port 988 [ 4065.751114] Key type lgssc registered [ 4066.097750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4068.820680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.979560] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.995031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.455292] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 4074.386432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.504596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.880300] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 4078.500663] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4080.513650] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4082.843947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.966786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.519715] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all 8 [ 4087.097752] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4096.479852] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.482748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.487174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4100.576879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4100.578449] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4100.582060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.375786] Lustre: server umount lustre-OST0000 complete [ 4105.183991] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4105.187717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.192585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.194929] Lustre: Skipped 1 previous similar message [ 4109.825528] Lustre: server umount lustre-MDT0000 complete [ 4110.816480] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.822058] LustreError: Skipped 1 previous similar message [ 4111.059124] LustreError: 233644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761325874 with bad export cookie 13211549827371176900 [ 4111.061129] LustreError: 166-1: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.063745] LustreError: 233644:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4111.209312] Lustre: server umount lustre-MDT0001 complete [ 4116.262380] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4117.378408] Key type lgssc unregistered [ 4117.494526] LNet: 237469:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4117.498506] LNet: Removed LNI 192.168.201.121@tcp [ 4117.823319] Key type .llcrypt unregistered [ 4117.824940] Key type ._llcrypt unregistered [ 4125.998037] alg: No test for adler32 (adler32-zlib) [ 4126.749365] Key type ._llcrypt registered [ 4126.750692] Key type .llcrypt registered [ 4126.790667] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4126.881937] Key type .llcrypt unregistered [ 4126.883742] Key type ._llcrypt unregistered [ 4134.827938] alg: No test for adler32 (adler32-zlib) [ 4135.580355] Key type ._llcrypt registered [ 4135.581702] Key type .llcrypt registered [ 4135.623235] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4141.520971] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4141.809801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4142.016329] Lustre: Lustre: Build Version: 2.15.7_12_gd220bd5 [ 4142.083237] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4142.085085] LNet: Accept secure, port 988 [ 4143.655077] Key type lgssc registered [ 4143.963306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4147.570990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.287374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4152.317974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4154.531979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4155.112346] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4003 sec ========== 13:11:57 (1761325917) [ 4168.569415] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4169.475802] Key type lgssc unregistered [ 4169.579364] LNet: 243688:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4169.584417] LNet: Removed LNI 192.168.201.121@tcp [ 4169.852630] Key type .llcrypt unregistered [ 4169.854107] Key type ._llcrypt unregistered