[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480119686 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006879] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009120] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.020170] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024788] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027115] *** VALIDATE tmpfs *** [ 0.028422] *** VALIDATE proc *** [ 0.030160] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035046] Spectre V2 : User space: Vulnerable [ 0.036015] Speculative Store Bypass: Vulnerable [ 0.039513] debug: unmapping init [mem 0xffffffff96459000-0xffffffff96460fff] [ 0.041296] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042862] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043032] ... version: 2 [ 0.044021] ... bit width: 48 [ 0.045079] ... generic registers: 4 [ 0.046019] ... value mask: 0000ffffffffffff [ 0.047023] ... max period: 00007fffffffffff [ 0.048040] ... fixed-purpose events: 3 [ 0.049021] ... event mask: 000000070000000f [ 0.051303] rcu: Hierarchical SRCU implementation. [ 0.053799] smp: Bringing up secondary CPUs ... [ 0.054757] x86: Booting SMP configuration: [ 0.055047] .... node #0, CPUs: #1 #2 #3 [ 0.059424] smp: Brought up 1 node, 4 CPUs [ 0.061028] smpboot: Max logical packages: 1 [ 0.062021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.257028] node 0 deferred pages initialised in 193ms [ 0.259186] devtmpfs: initialized [ 0.260289] x86/mm: Memory block size: 128MB [ 0.263313] gcov: version magic: 0x41383552 [ 0.265348] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.266141] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.267359] pinctrl core: initialized pinctrl subsystem [ 0.268259] [ 0.268708] ************************************************************* [ 0.269026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270033] ** ** [ 0.271020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.272020] ** ** [ 0.273019] ** This means that this kernel is built to expose internal ** [ 0.274014] ** IOMMU data structures, which may compromise security on ** [ 0.275012] ** your system. ** [ 0.276018] ** ** [ 0.277018] ** If you see this message and you are not debugging the ** [ 0.278019] ** kernel, report this immediately to your vendor! ** [ 0.279018] ** ** [ 0.280018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.281018] ************************************************************* [ 0.282793] NET: Registered protocol family 16 [ 0.283488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.284090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.285085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.286526] cpuidle: using governor menu [ 0.287000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.289672] PCI: Using configuration type 1 for base access [ 0.292144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.300126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.301038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.303050] cryptd: max_cpu_qlen set to 1000 [ 0.305376] ACPI: Added _OSI(Module Device) [ 0.306015] ACPI: Added _OSI(Processor Device) [ 0.307021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312302] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317070] ACPI: Interpreter enabled [ 0.319147] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320017] ACPI: Using IOAPIC for interrupt routing [ 0.322175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.325445] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.337499] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.340067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.342026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.346121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.351524] acpiphp: Slot [2] registered [ 0.353247] acpiphp: Slot [5] registered [ 0.354141] acpiphp: Slot [6] registered [ 0.356172] acpiphp: Slot [7] registered [ 0.358354] acpiphp: Slot [8] registered [ 0.360167] acpiphp: Slot [9] registered [ 0.362158] acpiphp: Slot [10] registered [ 0.364179] acpiphp: Slot [3] registered [ 0.366176] acpiphp: Slot [4] registered [ 0.367154] acpiphp: Slot [11] registered [ 0.369160] acpiphp: Slot [12] registered [ 0.371135] acpiphp: Slot [13] registered [ 0.373150] acpiphp: Slot [14] registered [ 0.374141] acpiphp: Slot [15] registered [ 0.376169] acpiphp: Slot [16] registered [ 0.378225] acpiphp: Slot [17] registered [ 0.380134] acpiphp: Slot [18] registered [ 0.381119] acpiphp: Slot [19] registered [ 0.383135] acpiphp: Slot [20] registered [ 0.385138] acpiphp: Slot [21] registered [ 0.387134] acpiphp: Slot [22] registered [ 0.388149] acpiphp: Slot [23] registered [ 0.390131] acpiphp: Slot [24] registered [ 0.392116] acpiphp: Slot [25] registered [ 0.394130] acpiphp: Slot [26] registered [ 0.396134] acpiphp: Slot [27] registered [ 0.397130] acpiphp: Slot [28] registered [ 0.399145] acpiphp: Slot [29] registered [ 0.401146] acpiphp: Slot [30] registered [ 0.403124] acpiphp: Slot [31] registered [ 0.404117] PCI host bridge to bus 0000:00 [ 0.406028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.410049] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.419035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.422035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.424212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.427172] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.431307] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.442831] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.447537] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.450029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.452025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.455023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459032] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.462113] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.468817] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.487903] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.493024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.499468] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.509023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.517022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.544023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.555287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.565020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.570020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.588021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.598042] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.607021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.635022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.645601] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.658020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.674022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.685851] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.691018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.701018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.724017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.735716] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.742025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.748023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.765025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.776567] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.779409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.783423] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.786392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.789273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.793241] iommu: Default domain type: Passthrough [ 0.795555] SCSI subsystem initialized [ 0.797219] ACPI: bus type USB registered [ 0.799160] usbcore: registered new interface driver usbfs [ 0.801113] usbcore: registered new interface driver hub [ 0.803117] usbcore: registered new device driver usb [ 0.805227] pps_core: LinuxPPS API ver. 1 registered [ 0.807016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.811089] PTP clock support registered [ 0.815050] EDAC MC: Ver: 3.0.0 [ 0.817203] PCI: Using ACPI for IRQ routing [ 0.819890] NetLabel: Initializing [ 0.821014] NetLabel: domain hash size = 128 [ 0.823014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826088] NetLabel: unlabeled traffic allowed by default [ 0.828145] vgaarb: loaded [ 0.830322] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.831018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.836401] clocksource: Switched to clocksource kvm-clock [ 0.951715] VFS: Disk quotas dquot_6.6.0 [ 0.953521] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.955712] *** VALIDATE ramfs *** [ 0.956791] *** VALIDATE hugetlbfs *** [ 0.958059] pnp: PnP ACPI init [ 0.960259] pnp: PnP ACPI: found 6 devices [ 0.995317] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.999172] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.001630] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.004018] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.006757] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.009503] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.012697] NET: Registered protocol family 2 [ 1.015545] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.021071] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.025287] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.031216] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.035767] TCP: Hash tables configured (established 65536 bind 65536) [ 1.040289] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.044196] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.047614] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050696] NET: Registered protocol family 1 [ 1.053893] RPC: Registered named UNIX socket transport module. [ 1.056656] RPC: Registered udp transport module. [ 1.058475] RPC: Registered tcp transport module. [ 1.060671] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.063702] NET: Registered protocol family 44 [ 1.065543] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.068268] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.070833] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.073800] PCI: CLS 0 bytes, default 64 [ 1.076427] Unpacking initramfs... [ 2.575290] debug: unmapping init [mem 0xffff8d01bcc54000-0xffff8d01bffbffff] [ 2.583203] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.585448] software IO TLB: mapped [mem 0x00000000a2c00000-0x00000000a6c00000] (64MB) [ 2.589297] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.117237] Initialise system trusted keyrings [ 3.118924] Key type blacklist registered [ 3.120595] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.132043] zbud: loaded [ 3.135418] *** VALIDATE nfs *** [ 3.137055] *** VALIDATE nfs4 *** [ 3.139033] pstore: using deflate compression [ 3.144038] Platform Keyring initialized [ 3.275168] NET: Registered protocol family 38 [ 3.277029] Key type asymmetric registered [ 3.278791] Asymmetric key parser 'x509' registered [ 3.281354] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.283965] io scheduler mq-deadline registered [ 3.285781] io scheduler kyber registered [ 3.287307] io scheduler bfq registered [ 3.288578] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.291782] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.295251] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.298616] ACPI: Power Button [PWRF] [ 3.304410] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.311928] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.328101] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.341694] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.361141] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.389898] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.420693] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.427727] Non-volatile memory driver v1.3 [ 3.429729] Linux agpgart interface v0.103 [ 3.467749] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.471024] vda: detected capacity change from 0 to 74760192 [ 3.487365] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.491098] vdb: detected capacity change from 0 to 1073741824 [ 3.513831] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.517368] vdc: detected capacity change from 0 to 2621440000 [ 3.535134] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.538926] vdd: detected capacity change from 0 to 2621440000 [ 3.554689] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.557381] vde: detected capacity change from 0 to 4294967296 [ 3.572512] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.576114] vdf: detected capacity change from 0 to 4294967296 [ 3.583359] libphy: Fixed MDIO Bus: probed [ 3.588500] usbcore: registered new interface driver usbserial_generic [ 3.590913] usbserial: USB Serial support registered for generic [ 3.593385] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.597719] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.599563] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.603744] mousedev: PS/2 mouse device common for all mice [ 3.606629] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.608678] rtc_cmos 00:05: RTC can wake from S4 [ 3.613591] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.613920] rtc_cmos 00:05: registered as rtc0 [ 3.618416] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.618465] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.621368] intel_pstate: CPU model not supported [ 3.627539] hid: raw HID events driver (C) Jiri Kosina [ 3.630051] usbcore: registered new interface driver usbhid [ 3.631802] usbhid: USB HID core driver [ 3.633061] drop_monitor: Initializing network drop monitor service [ 3.635184] Initializing XFRM netlink socket [ 3.636605] NET: Registered protocol family 10 [ 3.639368] Segment Routing with IPv6 [ 3.640977] NET: Registered protocol family 17 [ 3.643552] mpls_gso: MPLS GSO support [ 3.650885] RAS: Correctable Errors collector initialized. [ 3.653629] AVX version of gcm_enc/dec engaged. [ 3.655925] AES CTR mode by8 optimization enabled [ 3.739290] sched_clock: Marking stable (3739257896, 0)->(4729390283, -990132387) [ 3.743101] registered taskstats version 1 [ 3.745576] Loading compiled-in X.509 certificates [ 3.747544] zswap: loaded using pool lzo/zbud [ 3.777899] Key type big_key registered [ 3.791431] Key type encrypted registered [ 3.793079] ima: No TPM chip found, activating TPM-bypass! [ 3.795300] ima: Allocated hash algorithm: sha1 [ 3.797205] ima: No architecture policies found [ 3.799230] evm: Initialising EVM extended attributes: [ 3.801460] evm: security.selinux [ 3.803059] evm: security.ima [ 3.804309] evm: security.capability [ 3.805772] evm: HMAC attrs: 0x1 [ 3.808316] rtc_cmos 00:05: setting system clock to 2026-08-31 05:12:08 UTC (1788153128) [ 3.815245] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 3.818908] debug: unmapping init [mem 0xffffffff96182000-0xffffffff96458fff] [ 3.828199] Write protecting the kernel read-only data: 28672k [ 3.832170] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 3.835534] debug: unmapping init [mem 0xffffffff95114000-0xffffffff951fffff] [ 3.874608] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.885438] systemd[1]: Detected virtualization kvm. [ 3.887458] systemd[1]: Detected architecture x86-64. [ 3.889220] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.914946] systemd[1]: No hostname configured. [ 3.916658] systemd[1]: Set hostname to . [ 3.919218] random: systemd: uninitialized urandom read (16 bytes read) [ 3.922663] systemd[1]: Initializing machine ID from random generator. [ 4.059393] random: systemd: uninitialized urandom read (16 bytes read) [ 4.062437] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.068000] random: systemd: uninitialized urandom read (16 bytes read) [ 4.072276] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.079074] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.736697] device-mapper: uevent: version 1.0.3 [ 4.739363] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.478857] virtio_net virtio0 ens2: renamed from eth0 [ 5.493399] random: fast init done [ 5.589334] scsi host0: ata_piix [ 5.626263] scsi host1: ata_piix [ 5.630299] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.633871] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.357972] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.317550] random: crng init done [ 10.319833] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.827683] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.063212] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.361106] SELinux: Disabled at runtime. [ 12.421961] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.433016] systemd[1]: Detected virtualization kvm. [ 12.435119] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.985951] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.989940] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.995151] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.001314] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.006503] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.015489] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.026383] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 13.087818] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.417993] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.822459] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.844929] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.990748] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.005884] EDAC sbridge: Ver: 1.1.2 [ 15.597405] Key type dns_resolver registered [ 15.902954] NFS: Registering the id_resolver key type [ 15.905515] Key type id_resolver registered [ 15.907141] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg324-server login: [ 42.756453] libcfs: loading out-of-tree module taints kernel. [ 42.778459] Key type ._llcrypt registered [ 42.779820] Key type .llcrypt registered [ 42.833192] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 51.941979] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 52.575366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.585060] alg: No test for adler32 (adler32-zlib) [ 53.618433] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 53.990023] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 55.624464] Key type lgssc registered [ 57.473402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.343019] hrtimer: interrupt took 15348469 ns [ 79.157916] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 126.200616] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 141.110068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.180796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.499399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.528596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.653123] Lustre: lustre-MDT0000: new disk, initializing [ 142.757067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.796796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.567840] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.286555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.388202] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.409303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.413309] Lustre: Skipped 1 previous similar message [ 162.482839] Lustre: lustre-MDT0001: new disk, initializing [ 162.534194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.554676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.568132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 167.447097] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.799612] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.825226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.054741] Lustre: lustre-OST0000: new disk, initializing [ 184.060951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.064979] Lustre: 8424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.128499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.434701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.443987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.552973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 191.632433] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.414480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.617779] Lustre: lustre-OST0001: new disk, initializing [ 207.625635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.636277] Lustre: 9498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.712484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.518260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.538304] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.599917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 214.880608] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.644423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.625909] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.913115] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.175813] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing yml_node [ 249.392040] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 252.275799] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 254.953276] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 256.795505] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:16:19 EDT 2026 [ 274.948238] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 276.587200] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.393383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.394823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 295.428569] Lustre: Skipped 3 previous similar messages [ 295.438250] Lustre: Skipped 3 previous similar messages [ 299.096627] Lustre: server umount lustre-MDT0000 complete [ 302.953866] LustreError: 10314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153427 with bad export cookie 7243726202710841121 [ 302.961285] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 302.965372] LustreError: 10314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 305.632865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.647102] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.647114] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 305.647282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.686544] Lustre: Skipped 1 previous similar message [ 307.683240] LustreError: 8998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.683843] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.701237] LustreError: 8998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 307.706059] Lustre: Skipped 2 previous similar messages [ 309.414974] Lustre: server umount lustre-MDT0001 complete [ 314.184030] Lustre: server umount lustre-OST0000 complete [ 318.384926] Lustre: server umount lustre-OST0001 complete [ 326.234078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 336.761590] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 371.092867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.328908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.356393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.426927] Lustre: lustre-MDT0000: new disk, initializing [ 371.533696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.555978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.083047] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 386.803345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.916640] Lustre: 15053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.970445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.974206] Lustre: Skipped 1 previous similar message [ 387.029124] Lustre: lustre-MDT0001: new disk, initializing [ 387.164721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 387.211347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 387.223256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.555697] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 401.698643] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 407.430582] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.970761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.315646] Lustre: lustre-OST0000: new disk, initializing [ 419.319479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.328260] Lustre: 17025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.454963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.632625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 420.643571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.764332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.782099] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 435.556846] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 441.001319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 441.454577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 446.341917] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 446.733981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.557449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.579289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.586418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.604390] Lustre: Skipped 1 previous similar message [ 451.634304] Lustre: Skipped 1 previous similar message [ 456.674363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.689196] Lustre: Skipped 1 previous similar message [ 463.332154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 463.479838] Lustre: server umount lustre-OST0000 complete [ 468.961939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.972788] LustreError: Skipped 1 previous similar message [ 468.977974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.994669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 469.003932] Lustre: Skipped 2 previous similar messages [ 483.296200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.677160] Lustre: server umount lustre-MDT0000 complete [ 487.396673] LustreError: 16220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.424572] LustreError: 16220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 487.932057] LustreError: 15048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153612 with bad export cookie 7243726202710842647 [ 487.936679] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.946310] LustreError: 15048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 488.429136] Lustre: server umount lustre-MDT0001 complete [ 492.417492] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:20:14 (1788153614) [ 499.065220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.588812] LustreError: 19292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.685097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.523487] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 504.801516] LustreError: 19293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.308666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.248740] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 526.255655] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 532.897707] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 540.582106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 547.070466] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 555.823564] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 567.780233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 567.788342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.798340] Lustre: Skipped 1 previous similar message [ 567.820706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.823927] Lustre: Skipped 6 previous similar messages [ 573.222592] Lustre: server umount lustre-OST0000 complete [ 579.045441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.074831] Lustre: Skipped 2 previous similar messages [ 591.328251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 591.846931] Lustre: server umount lustre-MDT0000 complete [ 594.403727] LustreError: 19292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.443844] LustreError: 19292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 596.145417] LustreError: 19272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153720 with bad export cookie 7243726202710843536 [ 596.150341] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.159723] LustreError: 19272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 596.922585] Lustre: server umount lustre-MDT0001 complete [ 609.674925] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 613.318105] Key type lgssc unregistered [ 613.703167] LNet: 22830:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.712644] LNetError: 22830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.739316] LNet: Removed LNI 192.168.203.124@tcp [ 614.860268] Key type .llcrypt unregistered [ 614.862969] Key type ._llcrypt unregistered [ 638.965508] Key type ._llcrypt registered [ 638.968039] Key type .llcrypt registered [ 639.084523] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 640.156496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.310445] alg: No test for adler32 (adler32-zlib) [ 641.392409] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 641.753061] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 643.536373] Key type lgssc registered [ 644.739944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.309802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.344705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.913170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.717829] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 664.038347] LustreError: 24241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.054761] LustreError: 24241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 667.452781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.917601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.095683] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 682.883990] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 690.656614] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 698.110106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 698.504621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.598233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 705.372646] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 715.351716] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 735.714175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 735.730777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.771281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 738.821302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 738.863596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 738.895592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.547066] Lustre: server umount lustre-OST0000 complete [ 749.025398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.029893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.052426] Lustre: Skipped 1 previous similar message [ 749.069892] Lustre: Skipped 2 previous similar messages [ 752.320130] Lustre: server umount lustre-MDT0000 complete [ 754.145427] LustreError: 24971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 754.177807] LustreError: 24971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 756.619662] LustreError: 24224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153881 with bad export cookie 2275363026892377540 [ 756.627867] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.646845] LustreError: 24224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 756.905685] Lustre: server umount lustre-MDT0001 complete [ 775.965460] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 779.631388] Key type lgssc unregistered [ 780.074659] LNet: 28247:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.088495] LNetError: 28247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.104894] LNet: Removed LNI 192.168.203.124@tcp [ 781.251606] Key type .llcrypt unregistered [ 781.254100] Key type ._llcrypt unregistered [ 805.506358] Key type ._llcrypt registered [ 805.508444] Key type .llcrypt registered [ 805.597080] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 806.505911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.665555] alg: No test for adler32 (adler32-zlib) [ 807.821706] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 808.199402] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 809.920227] Key type lgssc registered [ 811.340933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 820.865978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 820.886910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.351444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.246103] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 832.483059] LustreError: 29663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 832.500781] LustreError: 29663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 837.297372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.604878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 837.861586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 842.430861] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 850.643784] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 857.048088] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 874.638900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.157036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.683315] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 892.631766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 892.693897] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 903.146948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 903.150754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.153110] LustreError: Skipped 1 previous similar message [ 903.156443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 903.167197] Lustre: Skipped 1 previous similar message [ 904.807463] Lustre: server umount lustre-OST0000 complete [ 909.281759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 909.290425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.314592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 909.327380] Lustre: Skipped 1 previous similar message [ 913.382045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.390169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.513851] Lustre: server umount lustre-MDT0000 complete [ 918.500790] LustreError: 29657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.532983] LustreError: 29657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 920.220951] LustreError: 29644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154044 with bad export cookie 7391177086900372068 [ 920.229511] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 920.614616] Lustre: server umount lustre-MDT0001 complete [ 932.726963] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 935.998922] Key type lgssc unregistered [ 936.361858] LNet: 33542:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.378779] LNetError: 33542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.428828] LNet: Removed LNI 192.168.203.124@tcp [ 937.574509] Key type .llcrypt unregistered [ 937.577807] Key type ._llcrypt unregistered [ 965.444387] Key type ._llcrypt registered [ 965.446781] Key type .llcrypt registered [ 965.558873] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 966.591650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.611073] alg: No test for adler32 (adler32-zlib) [ 967.748123] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 968.041215] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 969.696308] Key type lgssc registered [ 970.850221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 979.656541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 979.696252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.398797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.347116] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 991.718497] LustreError: 34957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 991.746474] LustreError: 34957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 995.163294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.735641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1000.444728] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1008.428812] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1015.185187] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1023.130876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1023.520911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.631684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1030.153210] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1038.018615] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1056.807410] Lustre: server umount lustre-OST0000 complete [ 1066.978320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1066.996463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1067.014968] Lustre: Skipped 1 previous similar message [ 1067.044334] Lustre: Skipped 1 previous similar message [ 1072.098983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.127024] Lustre: Skipped 1 previous similar message [ 1072.544394] Lustre: 34267:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154181/real 1788154181] req@00000000bc13bfd3 x1875015465203200/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788154197 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1072.600969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1073.186972] Lustre: 34270:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154181/real 1788154181] req@00000000e7eaf288 x1875015465202944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788154197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1073.827722] Lustre: server umount lustre-MDT0000 complete [ 1077.734474] LustreError: 35961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.736431] Lustre: 34267:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154186/real 1788154186] req@00000000f1476432 x1875015465204352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788154202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1077.781933] LustreError: 35961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1079.249243] LustreError: 34939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154203 with bad export cookie 17253747780619265565 [ 1079.267711] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.708636] Lustre: server umount lustre-MDT0001 complete [ 1092.109617] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1095.146310] Key type lgssc unregistered [ 1095.499909] LNet: 38518:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.508958] LNetError: 38518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.549908] LNet: Removed LNI 192.168.203.124@tcp [ 1096.573825] Key type .llcrypt unregistered [ 1096.576299] Key type ._llcrypt unregistered [ 1121.414700] Key type ._llcrypt registered [ 1121.419940] Key type .llcrypt registered [ 1121.539228] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1122.849389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1122.910723] alg: No test for adler32 (adler32-zlib) [ 1124.155255] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1124.439086] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1126.160267] Key type lgssc registered [ 1127.335565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1135.225670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1135.242923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.619885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.535396] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1146.854501] LustreError: 39932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.871721] LustreError: 39932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1151.105579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.527222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.359903] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1164.462337] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.761577] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.394446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.914548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.014851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1185.966268] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1194.606351] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1202.437354] Lustre: lustre-OST0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1202.658867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1202.667781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.757494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1206.761800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.770637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.776186] Lustre: Skipped 1 previous similar message [ 1208.168540] Lustre: server umount lustre-OST0000 complete [ 1232.356374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.401639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.408606] Lustre: Skipped 2 previous similar messages [ 1236.222279] Lustre: server umount lustre-MDT0000 complete [ 1237.479180] LustreError: 39932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.505128] LustreError: 39932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1240.602104] LustreError: 39913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154365 with bad export cookie 10741606936907037736 [ 1240.621950] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.625988] LustreError: 39913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1247.121561] Lustre: server umount lustre-MDT0001 complete [ 1258.292311] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1262.186057] Key type lgssc unregistered [ 1262.576582] LNet: 43637:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.589776] LNetError: 43637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.618779] LNet: Removed LNI 192.168.203.124@tcp [ 1263.952077] Key type .llcrypt unregistered [ 1263.955948] Key type ._llcrypt unregistered [ 1292.927433] Key type ._llcrypt registered [ 1292.929350] Key type .llcrypt registered [ 1293.034607] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1294.213413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.439696] alg: No test for adler32 (adler32-zlib) [ 1295.514168] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1295.791768] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1297.512343] Key type lgssc registered [ 1299.129916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1307.691736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1307.708085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.233379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.086679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1318.378149] LustreError: 45067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.427773] LustreError: 45067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1323.336883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.593923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1328.780988] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1338.031327] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1345.087713] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1352.609193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1353.098587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.102245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1359.773055] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1368.988808] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1376.013501] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1378.786375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.794816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.803908] Lustre: Skipped 2 previous similar messages [ 1378.819333] Lustre: Skipped 2 previous similar messages [ 1381.126651] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 1381.623917] Lustre: server umount lustre-MDT0000 complete [ 1383.925592] LustreError: 45066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.963588] LustreError: 45066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1385.964582] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154510 with bad export cookie 111258971556881078 [ 1385.969708] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.976331] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1386.229077] LustreError: 45061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.254716] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 1389.025182] LustreError: 45062:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.027536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.342827] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 1391.355789] Lustre: Skipped 2 previous similar messages [ 1392.520496] Lustre: server umount lustre-MDT0001 complete [ 1414.471973] Lustre: server umount lustre-OST0000 complete [ 1437.333607] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1441.048422] Key type lgssc unregistered [ 1441.597439] LNet: 48930:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.613183] LNetError: 48930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.650902] LNet: Removed LNI 192.168.203.124@tcp [ 1443.805086] Key type .llcrypt unregistered [ 1443.808822] Key type ._llcrypt unregistered [ 1474.176225] Key type ._llcrypt registered [ 1474.182379] Key type .llcrypt registered [ 1474.324861] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1475.553854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.726790] alg: No test for adler32 (adler32-zlib) [ 1476.963343] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1477.374866] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1479.152654] Key type lgssc registered [ 1480.847901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1490.503802] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1490.539424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.208658] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1511.328299] LustreError: 50317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 1538.017185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.277037] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1554.967225] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1721.064109] Lustre: server umount lustre-OST0000 complete [ 1741.400372] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1744.355703] Key type lgssc unregistered [ 1744.751346] LNet: 51996:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.771965] LNetError: 51996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.788844] LNet: Removed LNI 192.168.203.124@tcp [ 1745.917390] Key type .llcrypt unregistered [ 1745.925264] Key type ._llcrypt unregistered [ 1771.863633] Key type ._llcrypt registered [ 1771.866469] Key type .llcrypt registered [ 1771.965446] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1773.531894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.582612] alg: No test for adler32 (adler32-zlib) [ 1774.691679] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1775.034624] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1776.729856] Key type lgssc registered [ 1778.117858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.048492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.071323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.620227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.363802] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1799.147220] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.177935] LustreError: 53414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1803.085876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.520369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1809.060815] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1817.699537] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1824.377156] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1831.817193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1832.172497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.237022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1838.821604] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1847.312201] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.931900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1853.932098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.943098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.956975] Lustre: Skipped 1 previous similar message [ 1853.974220] Lustre: Skipped 1 previous similar message [ 1856.489845] Lustre: server umount lustre-OST0000 complete [ 1864.165858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.194044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.202546] Lustre: Skipped 1 previous similar message [ 1867.181517] Lustre: server umount lustre-MDT0000 complete [ 1869.285848] LustreError: 53413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.296595] LustreError: 53413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1872.014536] LustreError: 53394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154996 with bad export cookie 687713053633731489 [ 1872.022580] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.036887] LustreError: 53394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1872.482148] Lustre: server umount lustre-MDT0001 complete [ 1885.509843] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1888.859659] Key type lgssc unregistered [ 1889.229785] LNet: 56945:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.233753] LNetError: 56945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.278650] LNet: Removed LNI 192.168.203.124@tcp [ 1891.245540] Key type .llcrypt unregistered [ 1891.247152] Key type ._llcrypt unregistered [ 1919.314779] Key type ._llcrypt registered [ 1919.319995] Key type .llcrypt registered [ 1919.422205] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1919.980068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.017360] alg: No test for adler32 (adler32-zlib) [ 1921.153244] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1921.489626] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1923.176557] Key type lgssc registered [ 1924.460763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1932.798708] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1932.827140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.384382] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1953.504827] LustreError: 58332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 1980.256299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.365114] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1994.969514] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2001.760596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.851714] LustreError: 57668:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000e61fd0e x1875016465516800/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.269493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.349951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2006.665543] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2007.521461] LustreError: 58357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.566286] LustreError: 58357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2012.643400] LustreError: 58358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.695825] LustreError: 58357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.220281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.530058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2021.107879] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2029.194992] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.128974] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2038.242828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2038.251331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.283704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.338240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2042.352312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2042.360939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.662208] Lustre: server umount lustre-OST0000 complete [ 2067.938968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2067.940324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.956138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.966199] Lustre: Skipped 1 previous similar message [ 2067.980572] Lustre: Skipped 2 previous similar messages [ 2073.065568] LustreError: 59373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.085893] LustreError: 59373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2073.204511] Lustre: server umount lustre-MDT0000 complete [ 2077.363949] LustreError: 58338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155202 with bad export cookie 11303795711083784200 [ 2077.368305] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2077.394111] LustreError: 58338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2083.972959] Lustre: server umount lustre-MDT0001 complete [ 2096.482101] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2099.858760] Key type lgssc unregistered [ 2100.249895] LNet: 62081:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.271674] LNetError: 62081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.361974] LNet: Removed LNI 192.168.203.124@tcp [ 2102.348923] Key type .llcrypt unregistered [ 2102.363899] Key type ._llcrypt unregistered [ 2130.042126] Key type ._llcrypt registered [ 2130.046791] Key type .llcrypt registered [ 2130.188643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2131.351755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2131.396216] alg: No test for adler32 (adler32-zlib) [ 2132.472869] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2132.729328] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2134.424162] Key type lgssc registered [ 2135.676598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2144.455437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2144.478730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.989433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.730901] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2155.504659] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.524290] LustreError: 63499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2160.425453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.634876] LustreError: 63500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.095941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.455734] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2175.212784] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.848946] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.743430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2191.227091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.027716] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2208.044924] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2215.824695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2231.265412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2231.273933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.278634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.295492] Lustre: Skipped 1 previous similar message [ 2232.537252] Lustre: server umount lustre-OST0000 complete [ 2236.386368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.396570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.398210] Lustre: Skipped 1 previous similar message [ 2236.400915] Lustre: Skipped 1 previous similar message [ 2241.508470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.519708] Lustre: Skipped 2 previous similar messages [ 2246.627396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.635391] Lustre: Skipped 1 previous similar message [ 2250.208393] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2250.505647] Lustre: server umount lustre-MDT0000 complete [ 2251.751158] LustreError: 63500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.764563] LustreError: 63500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2254.159735] LustreError: 63480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155378 with bad export cookie 5048329673876056312 [ 2254.163846] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.177888] LustreError: 63480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2254.492957] Lustre: server umount lustre-MDT0001 complete [ 2265.727704] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2268.548639] Key type lgssc unregistered [ 2268.894544] LNet: 67060:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.905813] LNetError: 67060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.934278] LNet: Removed LNI 192.168.203.124@tcp [ 2271.049719] Key type .llcrypt unregistered [ 2271.051932] Key type ._llcrypt unregistered [ 2298.251637] Key type ._llcrypt registered [ 2298.261959] Key type .llcrypt registered [ 2298.360233] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2299.194693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2299.224829] alg: No test for adler32 (adler32-zlib) [ 2300.344711] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2300.619184] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2302.304178] Key type lgssc registered [ 2303.243728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2310.852752] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:50:33 (1788155433) [ 2316.312843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2316.330692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.698020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.627305] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2328.039965] LustreError: 68764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.057018] LustreError: 68764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2328.546238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.869965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2332.861415] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2339.889196] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2344.877255] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2350.822632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2351.388958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.078294] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2362.678694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2364.441450] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2370.732590] Lustre: Failing over lustre-MDT0000 [ 2370.978812] Lustre: server umount lustre-MDT0000 complete [ 2372.367757] LustreError: 68760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.100231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.117124] Lustre: Skipped 1 previous similar message [ 2377.471019] LustreError: 68760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.511083] LustreError: 68760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2381.229386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.327656] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.470391] Lustre: *** cfs_fail_loc=135, val=0*** [ 2381.557766] LustreError: 71849:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2381.570890] LustreError: 71849:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.124@tcp: cfg command failed: rc = -2 [ 2381.585657] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2381.585657] [ 2381.601181] LustreError: MGC192.168.203.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2381.633687] LustreError: 71838:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2381.643200] LustreError: 71838:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 2381.652896] LustreError: 71838:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2381.658758] LustreError: 71838:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2381.686930] Lustre: server umount lustre-MDT0000 complete [ 2381.692764] LustreError: 71838:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2382.573777] LustreError: 69815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.602904] LustreError: 69815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2387.696752] LustreError: 68758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.738520] LustreError: 68758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2389.061738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.195425] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.423826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.503043] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2392.754704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2393.773908] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2394.605460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2394.623907] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2394.695142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2404.853505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.854333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.865925] Lustre: Skipped 1 previous similar message [ 2404.866717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.961187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.972763] Lustre: Skipped 2 previous similar messages [ 2415.072117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2415.077298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.093528] Lustre: Skipped 1 previous similar message [ 2415.270459] Lustre: server umount lustre-OST0000 complete [ 2420.193289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2420.200358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.206165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.223297] Lustre: Skipped 2 previous similar messages [ 2425.319580] LustreError: 68758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.338451] LustreError: 68758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2425.418931] Lustre: server umount lustre-MDT0000 complete [ 2428.982334] LustreError: 68745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155553 with bad export cookie 9330120923821205352 [ 2428.993920] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2428.995601] LustreError: 68745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2429.298648] Lustre: server umount lustre-MDT0001 complete [ 2441.304536] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2444.035724] Key type lgssc unregistered [ 2444.314619] LNet: 73629:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.321553] LNetError: 73629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.335872] LNet: Removed LNI 192.168.203.124@tcp [ 2445.427191] Key type .llcrypt unregistered [ 2445.430403] Key type ._llcrypt unregistered [ 2472.218037] Key type ._llcrypt registered [ 2472.220889] Key type .llcrypt registered [ 2472.346659] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2473.804700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.837953] alg: No test for adler32 (adler32-zlib) [ 2475.060789] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2475.289883] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2476.992284] Key type lgssc registered [ 2477.950643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2487.922934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2487.954744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.796550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.059488] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2500.070870] LustreError: 75044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.126170] LustreError: 75044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2505.187110] LustreError: 75043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.754414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.198975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2514.053432] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2524.425387] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2531.774794] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2538.172215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2538.500690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.582605] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2552.035687] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2559.995847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2566.935429] Lustre: Failing over lustre-MDT0000 [ 2567.170425] Lustre: server umount lustre-MDT0000 complete [ 2567.958651] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.211210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.240181] LustreError: 75039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.267486] LustreError: 75039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2575.334240] LustreError: 75039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.357835] LustreError: 75039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2576.706910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.836413] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.020431] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2577.137297] LustreError: 78152:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2577.143586] LustreError: 78152:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.124@tcp: cfg command failed: rc = -9 [ 2577.155656] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2577.155656] [ 2577.169475] LustreError: MGC192.168.203.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2577.185652] LustreError: 78140:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2577.194797] LustreError: 78140:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -9 [ 2577.206051] LustreError: 78140:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2577.216569] LustreError: 78140:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2577.247140] Lustre: server umount lustre-MDT0000 complete [ 2577.250046] LustreError: 78140:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2585.444159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.549581] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.557936] LustreError: 75035:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2585.572966] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.596359] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2585.875196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.937322] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2588.401644] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2590.373333] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2591.250650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2591.314149] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2591.409735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2599.905851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2599.916527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.935021] Lustre: Skipped 2 previous similar messages [ 2599.946095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.447198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.466342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.623190] Lustre: server umount lustre-OST0000 complete [ 2611.688645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.698721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.709658] Lustre: Skipped 1 previous similar message [ 2611.732164] Lustre: Skipped 2 previous similar messages [ 2614.428805] Lustre: server umount lustre-MDT0000 complete [ 2616.801673] LustreError: 75044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2617.452474] LustreError: 75024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155742 with bad export cookie 9462743628894543183 [ 2617.457928] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2617.471419] LustreError: 75024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2617.729353] Lustre: server umount lustre-MDT0001 complete [ 2627.476195] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2629.809646] Key type lgssc unregistered [ 2630.072610] LNet: 79935:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.079281] LNetError: 79935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.104625] LNet: Removed LNI 192.168.203.124@tcp [ 2630.892589] Key type .llcrypt unregistered [ 2630.895959] Key type ._llcrypt unregistered [ 2643.632597] Key type ._llcrypt registered [ 2643.638098] Key type .llcrypt registered [ 2643.771224] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2644.048692] Key type .llcrypt unregistered [ 2644.052935] Key type ._llcrypt unregistered [ 2666.508177] Key type ._llcrypt registered [ 2666.511744] Key type .llcrypt registered [ 2666.626170] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2667.632811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2667.698713] alg: No test for adler32 (adler32-zlib) [ 2668.789469] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2669.100104] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2670.824398] Key type lgssc registered [ 2671.922815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2679.553497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2679.591058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.073715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.509687] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2690.534794] LustreError: 81843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.555192] LustreError: 81843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2694.010799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.294797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2698.881394] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2706.272358] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2711.667532] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2717.760503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.088328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.874464] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2726.332951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2731.598390] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2739.686371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2739.698689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.723944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.730966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.731921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.736967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.774555] Lustre: Skipped 1 previous similar message [ 2745.945919] Lustre: server umount lustre-OST0000 complete [ 2750.948766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2750.956442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.966453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2755.858730] Lustre: server umount lustre-MDT0000 complete [ 2757.090252] LustreError: 81840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.109934] LustreError: 81840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2759.348199] LustreError: 81823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155884 with bad export cookie 4313043495787695628 [ 2759.354696] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.364361] LustreError: 81823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2759.953857] Lustre: server umount lustre-MDT0001 complete [ 2771.174931] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2774.011957] Key type lgssc unregistered [ 2774.333767] LNet: 85375:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2774.339233] LNetError: 85375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2774.356742] LNet: Removed LNI 192.168.203.124@tcp [ 2775.450210] Key type .llcrypt unregistered [ 2775.452086] Key type ._llcrypt unregistered [ 2800.190996] Key type ._llcrypt registered [ 2800.192784] Key type .llcrypt registered [ 2800.268479] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2801.104761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2801.120946] alg: No test for adler32 (adler32-zlib) [ 2802.198853] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2802.385888] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2804.032172] Key type lgssc registered [ 2805.072451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2812.852268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2812.867055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.349633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.472344] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2824.678046] LustreError: 86789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.693212] LustreError: 86789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2826.000608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.424110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2830.795501] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2838.219669] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2844.127595] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2850.167102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.460816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.553381] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2859.719150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2865.136482] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2872.293762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2872.303281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.316374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2873.343684] Lustre: server umount lustre-OST0000 complete [ 2877.409949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.423954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.448183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.485491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.485890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.527027] Lustre: Skipped 1 previous similar message [ 2885.604773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.614720] Lustre: Skipped 1 previous similar message [ 2886.700402] Lustre: server umount lustre-MDT0000 complete [ 2890.721777] LustreError: 87517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.760219] LustreError: 87517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2891.058231] LustreError: 86769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156015 with bad export cookie 6884253634627605419 [ 2891.060780] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2891.075051] LustreError: 86769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2891.476302] Lustre: server umount lustre-MDT0001 complete [ 2903.946389] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2906.684519] Key type lgssc unregistered [ 2907.009582] LNet: 90319:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.021716] LNetError: 90319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.038250] LNet: Removed LNI 192.168.203.124@tcp [ 2907.920653] Key type .llcrypt unregistered [ 2907.923557] Key type ._llcrypt unregistered [ 2931.329994] Key type ._llcrypt registered [ 2931.342989] Key type .llcrypt registered [ 2931.480786] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2932.752401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2932.808562] alg: No test for adler32 (adler32-zlib) [ 2933.818845] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2934.098484] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2935.801257] Key type lgssc registered [ 2937.132169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2945.747932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2945.778264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.350959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.896395] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2956.774031] LustreError: 91746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.788498] LustreError: 91746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2961.896432] LustreError: 91747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2962.491969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.803596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2967.550428] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2976.596796] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2982.898864] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2989.693398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.060671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.132059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2996.603601] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3004.312216] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3021.809146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3021.823724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.842957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.817205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3022.831615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3022.836639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.564503] Lustre: server umount lustre-OST0000 complete [ 3033.059136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.068233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.082272] Lustre: Skipped 1 previous similar message [ 3033.090827] Lustre: Skipped 2 previous similar messages [ 3037.932973] Lustre: server umount lustre-MDT0000 complete [ 3038.179418] LustreError: 91741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.201446] LustreError: 91741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3041.457888] LustreError: 94247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156166 with bad export cookie 1956736417440111656 [ 3041.461333] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.486767] LustreError: 94247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3041.945631] Lustre: server umount lustre-MDT0001 complete [ 3051.881345] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3054.182078] Key type lgssc unregistered [ 3054.407186] LNet: 95301:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.415805] LNetError: 95301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3054.429346] LNet: Removed LNI 192.168.203.124@tcp [ 3055.171526] Key type .llcrypt unregistered [ 3055.180184] Key type ._llcrypt unregistered [ 3074.126394] Key type ._llcrypt registered [ 3074.129806] Key type .llcrypt registered [ 3074.242551] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3075.110019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3075.212813] alg: No test for adler32 (adler32-zlib) [ 3076.233358] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3076.451092] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3078.152184] Key type lgssc registered [ 3079.154994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3086.237981] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3086.262592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.792406] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3106.912971] LustreError: 96667:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 3133.536215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.197956] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3146.648995] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3152.339928] Lustre: server umount lustre-OST0000 complete [ 3160.847639] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:04:43 (1788156283) [ 3167.786775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.191767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3172.238369] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3173.346218] LustreError: 98307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.375158] LustreError: 98307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3177.443800] LustreError: 98308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.287594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.804864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3186.352804] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3195.318417] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3201.063407] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3208.053952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.359726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.511450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3214.819719] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3222.373316] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3237.350825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3237.351071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.374044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3237.377742] Lustre: Skipped 1 previous similar message [ 3242.467180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3242.481779] Lustre: Skipped 2 previous similar messages [ 3243.031567] Lustre: server umount lustre-OST0000 complete [ 3247.588035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.628530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.642177] Lustre: Skipped 1 previous similar message [ 3252.708818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.723734] Lustre: Skipped 1 previous similar message [ 3252.923022] Lustre: server umount lustre-MDT0000 complete [ 3256.317753] LustreError: 98288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156381 with bad export cookie 1248374839703146445 [ 3256.327133] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.331089] LustreError: 98288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3256.626223] Lustre: server umount lustre-MDT0001 complete [ 3266.474387] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3268.949300] Key type lgssc unregistered [ 3269.241786] LNet: 102000:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3269.253799] LNetError: 102000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3269.266981] LNet: Removed LNI 192.168.203.124@tcp [ 3270.165121] Key type .llcrypt unregistered [ 3270.167877] Key type ._llcrypt unregistered [ 3310.812914] Key type ._llcrypt registered [ 3310.815842] Key type .llcrypt registered [ 3310.967966] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3312.141319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3312.211331] alg: No test for adler32 (adler32-zlib) [ 3313.377904] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3313.761886] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3315.424637] Key type lgssc registered [ 3316.622787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3325.920798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3325.946904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.605820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.603541] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3338.212923] LustreError: 104187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.239128] LustreError: 104187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3343.333804] LustreError: 104188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.374245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.895493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3349.566840] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3358.706808] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3364.783533] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3371.822859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.216542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.236913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3378.332769] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3384.547318] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3398.125168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3398.136197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.150983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3399.137075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3399.143199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3399.145652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.160876] Lustre: Skipped 1 previous similar message [ 3403.021671] Lustre: server umount lustre-OST0000 complete [ 3409.381249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.384065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.391925] Lustre: Skipped 1 previous similar message [ 3412.363368] Lustre: server umount lustre-MDT0000 complete [ 3414.497779] LustreError: 106680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3414.520743] LustreError: 106680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3415.930745] LustreError: 104168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156540 with bad export cookie 7661005707122302696 [ 3415.935689] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3415.948143] LustreError: 104168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3416.295061] Lustre: server umount lustre-MDT0001 complete [ 3426.363523] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3429.016433] Key type lgssc unregistered [ 3429.343633] LNet: 107743:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3429.350884] LNetError: 107743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3429.365212] LNet: Removed LNI 192.168.203.124@tcp [ 3430.018218] Key type .llcrypt unregistered [ 3430.022456] Key type ._llcrypt unregistered [ 3447.361651] Key type ._llcrypt registered [ 3447.364438] Key type .llcrypt registered [ 3447.489586] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3448.457282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3448.488017] alg: No test for adler32 (adler32-zlib) [ 3449.556086] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3449.761487] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3451.464181] Key type lgssc registered [ 3452.640778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3461.995977] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3462.032367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3477.666877] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3482.784798] LustreError: 108987:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 3509.472220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.566150] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3521.613555] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3527.824922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.087476] LustreError: MGC192.168.203.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3528.102381] LustreError: 110010:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3528.111663] LustreError: 110010:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 3528.118818] LustreError: 110010:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 3528.203646] Lustre: server umount lustre-MDT0000 complete [ 3528.205874] LustreError: 110010:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3548.640315] Lustre: 108325:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156657/real 1788156657] req@000000000eafb3fe x1875018067741952/t0(0) o400->MGC192.168.203.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788156673 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3548.678182] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3548.699372] LustreError: 108999:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3548.849866] Lustre: server umount lustre-OST0000 complete [ 3557.926139] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3569.031167] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3604.047270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.261273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3604.292252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3604.375305] Lustre: lustre-MDT0000: new disk, initializing [ 3604.439634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.464083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3608.548761] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3618.824411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.963198] Lustre: 112800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3619.022710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3619.026579] Lustre: Skipped 1 previous similar message [ 3619.111236] Lustre: lustre-MDT0001: new disk, initializing [ 3619.203123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3619.258775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3619.269666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3623.179354] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3632.451692] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3638.617959] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3646.472059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.818461] Lustre: lustre-OST0000: new disk, initializing [ 3646.822895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3646.841716] Lustre: 114768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3647.000340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.867891] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3653.704208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.717668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3653.886749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3664.580719] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3670.263466] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3670.589911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3674.675875] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3674.926782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3679.714706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3679.714997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.725671] LustreError: Skipped 1 previous similar message [ 3679.729226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3679.752810] Lustre: Skipped 1 previous similar message [ 3683.384628] Lustre: server umount lustre-OST0000 complete [ 3689.956942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.986537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.996090] Lustre: Skipped 2 previous similar messages [ 3693.970971] Lustre: server umount lustre-MDT0000 complete [ 3697.912232] LustreError: 113603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156822 with bad export cookie 16135255421242543496 [ 3697.914342] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.932431] LustreError: 113603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3698.205764] Lustre: server umount lustre-MDT0001 complete [ 3708.056343] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:13:50 (1788156830) [ 3711.948576] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3732.005945] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3743.479214] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3781.831157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3782.075908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3782.110307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3782.166927] Lustre: lustre-MDT0000: new disk, initializing [ 3782.282184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.301633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3786.482927] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3797.853884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.999714] Lustre: 119671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3798.064607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3798.071402] Lustre: Skipped 1 previous similar message [ 3798.136943] Lustre: lustre-MDT0001: new disk, initializing [ 3798.230385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.272523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3798.287458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3802.710487] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3812.986317] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3818.943770] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3825.527159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.913722] Lustre: lustre-OST0000: new disk, initializing [ 3825.922269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3825.933939] Lustre: 121638:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3826.071751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.429874] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3833.934802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3833.955101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3834.089891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3842.393169] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3847.494705] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3847.758097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3853.744654] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3854.062270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3859.938015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3859.942505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.948789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3859.949478] Lustre: Skipped 2 previous similar messages [ 3862.496442] Lustre: server umount lustre-OST0000 complete [ 3870.188979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3870.194068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3870.204879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3870.204888] Lustre: Skipped 1 previous similar message [ 3870.241943] Lustre: Skipped 1 previous similar message [ 3872.852230] LustreError: 123179:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000a9d45af6) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3872.964731] Lustre: server umount lustre-MDT0000 complete [ 3875.297535] LustreError: 119683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.332240] LustreError: 119683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3877.664068] LustreError: 119662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157002 with bad export cookie 16135255421242544385 [ 3877.670717] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3877.683436] LustreError: 119662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3878.170772] Lustre: server umount lustre-MDT0001 complete [ 3891.926959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.053270] LustreError: 123857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3893.357627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.346029] LustreError: 123858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.737861] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3902.440048] LustreError: 123857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.555311] LustreError: 123858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.325371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.614068] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3923.926974] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.301510] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3936.029556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3942.975736] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3951.811697] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3960.326198] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3968.495073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3968.516580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.564923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3968.586577] Lustre: Skipped 1 previous similar message [ 3979.748881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3979.757653] Lustre: Skipped 5 previous similar messages [ 3981.281401] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3981.700932] Lustre: server umount lustre-OST0000 complete [ 3988.452114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.475098] LustreError: Skipped 1 previous similar message [ 3988.479976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.501728] Lustre: Skipped 1 previous similar message [ 3992.893389] Lustre: server umount lustre-MDT0000 complete [ 3995.112614] LustreError: 123857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.131388] LustreError: 123857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3996.967986] LustreError: 123838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157121 with bad export cookie 16135255421242545267 [ 3996.981392] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.984840] LustreError: 123838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3997.317871] Lustre: server umount lustre-MDT0001 complete [ 4010.812870] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4013.858499] Key type lgssc unregistered [ 4014.229699] LNet: 127486:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4014.241583] LNetError: 127486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4015.271684] LNet: Removed LNI 192.168.203.124@tcp [ 4016.384491] Key type .llcrypt unregistered [ 4016.386915] Key type ._llcrypt unregistered [ 4038.295831] Key type ._llcrypt registered [ 4038.300522] Key type .llcrypt registered [ 4038.444545] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4056.333814] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4057.752868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4057.804035] alg: No test for adler32 (adler32-zlib) [ 4059.028465] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4059.325985] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4060.984734] Key type lgssc registered [ 4062.634737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4102.784103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4102.822253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.134323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4104.169919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4104.290833] Lustre: lustre-MDT0000: new disk, initializing [ 4104.448664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.480630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4109.636658] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4121.112851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.252797] Lustre: 130647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4121.324761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4121.331565] Lustre: Skipped 1 previous similar message [ 4121.461145] Lustre: lustre-MDT0001: new disk, initializing [ 4121.576671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4121.636052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4121.657727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4127.053501] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4138.088086] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4144.766676] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.168565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4152.440337] Lustre: lustre-OST0000: new disk, initializing [ 4152.444325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4152.450185] Lustre: 132614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4152.541776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.736411] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4162.095583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4162.107334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4162.223442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4170.680521] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4175.684787] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4176.013643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4180.841726] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4181.087362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4188.139634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4188.148848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.153258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.166195] Lustre: Skipped 1 previous similar message [ 4188.183147] Lustre: Skipped 1 previous similar message [ 4193.250596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4193.267284] Lustre: Skipped 1 previous similar message [ 4197.344141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4197.601713] Lustre: server umount lustre-OST0000 complete [ 4203.490534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4203.490973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4203.504527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4203.509114] Lustre: Skipped 1 previous similar message [ 4203.556134] LustreError: Skipped 1 previous similar message [ 4208.611435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.622960] Lustre: Skipped 2 previous similar messages [ 4213.731743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.745061] Lustre: Skipped 1 previous similar message [ 4216.288441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4216.661744] Lustre: server umount lustre-MDT0000 complete [ 4222.193679] LustreError: 130640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157346 with bad export cookie 8644252011880101364 [ 4222.226662] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.796837] Lustre: server umount lustre-MDT0001 complete [ 4233.627581] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:22:36 (1788157356) [ 4242.453683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4243.048855] LustreError: 135028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4243.075920] LustreError: 135028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4243.172635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.889389] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4248.549515] LustreError: 135029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4252.663865] LustreError: 135028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.789430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.220954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4262.995270] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4272.654492] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.316251] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4282.061471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.077885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.092706] Lustre: Skipped 1 previous similar message [ 4288.470561] Lustre: server umount lustre-MDT0000 complete [ 4292.583562] LustreError: 135971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.617309] LustreError: 135971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4292.741541] LustreError: 135009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157417 with bad export cookie 8644252011880102246 [ 4292.761078] LustreError: 135009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4292.770150] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4293.350413] Lustre: server umount lustre-MDT0001 complete [ 4302.112890] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:23:44 (1788157424) [ 4309.237616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4324.832638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4329.952648] LustreError: 137445:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 4355.681447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.423526] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4371.088521] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4374.180934] Lustre: server umount lustre-OST0000 complete [ 4384.425932] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:25:06 (1788157506) [ 4392.218896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.935148] LustreError: 138892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.001488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4398.802731] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4403.180292] LustreError: 138893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.218557] LustreError: 138893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4409.002321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.651070] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4423.114266] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4429.800276] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4437.440167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4437.778826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.783489] Lustre: Skipped 1 previous similar message [ 4446.088731] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4456.594936] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4470.757704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.763047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.764458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4470.781839] Lustre: Skipped 1 previous similar message [ 4470.796601] Lustre: Skipped 5 previous similar messages [ 4472.750852] Lustre: server umount lustre-MDT0000 complete [ 4475.873775] LustreError: 138892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.894782] LustreError: 138892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4476.577347] LustreError: 138874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157601 with bad export cookie 8644252011880102911 [ 4476.579044] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.596312] LustreError: 138874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4476.928660] Lustre: server umount lustre-MDT0001 complete [ 4490.751596] Lustre: server umount lustre-OST0000 complete [ 4500.236514] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:27:02 (1788157622) [ 4506.631970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.315860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.169345] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4512.751112] LustreError: 142548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.772650] LustreError: 142548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4523.364959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.582352] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4538.354059] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.407540] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4550.967163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4552.384384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4557.526992] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4564.863359] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4569.947494] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4570.207599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4575.291249] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4575.552380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4578.276370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.285113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.321759] Lustre: Skipped 1 previous similar message [ 4578.325491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.568168] Lustre: server umount lustre-OST0000 complete [ 4589.541178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.552776] Lustre: Skipped 2 previous similar messages [ 4594.498330] Lustre: server umount lustre-MDT0000 complete [ 4594.660654] LustreError: 142549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.670202] LustreError: 142549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4597.975142] LustreError: 142529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157722 with bad export cookie 8644252011880103982 [ 4597.976887] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4597.989562] LustreError: 142529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4598.372777] Lustre: server umount lustre-MDT0001 complete [ 4607.801709] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:28:49 (1788157729) [ 4615.302056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4630.816653] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4635.937408] LustreError: 146708:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 4661.665500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4661.670568] Lustre: Skipped 2 previous similar messages [ 4668.339661] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4676.704584] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4684.055739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.884899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4689.959475] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4699.723904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.134851] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4715.062931] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4721.338917] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4726.587365] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4726.832747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4732.465889] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4732.721431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4736.486744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.488377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4736.492730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.492739] Lustre: Skipped 4 previous similar messages [ 4736.522099] Lustre: Skipped 1 previous similar message [ 4736.559964] LustreError: Skipped 1 previous similar message [ 4749.280146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4749.482507] Lustre: server umount lustre-MDT0000 complete [ 4751.845619] LustreError: 146734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.876428] LustreError: 146734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4753.735313] LustreError: 146714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157878 with bad export cookie 8644252011880105004 [ 4753.757601] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4753.760088] LustreError: 146714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4754.194049] Lustre: server umount lustre-MDT0001 complete [ 4767.719210] Lustre: server umount lustre-OST0000 complete [ 4775.939760] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:31:38 (1788157898) [ 4782.799935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.496543] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4803.616450] LustreError: 150906:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 4835.674344] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4843.376812] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.054904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.520859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4854.635250] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4863.303872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.343604] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4876.468476] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4882.131750] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4889.526468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4889.829188] Lustre: lustre-OST0001: new disk, initializing [ 4889.834891] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4889.842603] Lustre: 153778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4896.676618] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4898.881693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4898.900631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4899.037305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4906.826528] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4912.371446] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4912.654689] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4917.444176] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4917.660632] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4920.291065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4920.306333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.327758] Lustre: Skipped 1 previous similar message [ 4920.334939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4920.342165] Lustre: Skipped 8 previous similar messages [ 4934.112234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4934.284551] Lustre: server umount lustre-OST0000 complete [ 4940.268515] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.280927] LustreError: Skipped 1 previous similar message [ 4944.329921] Lustre: server umount lustre-OST0001 complete [ 4954.677820] Lustre: server umount lustre-MDT0000 complete [ 4958.631590] LustreError: 150913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158083 with bad export cookie 8644252011880105956 [ 4958.633801] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4958.644752] LustreError: 150913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4959.098750] Lustre: server umount lustre-MDT0001 complete [ 4984.102379] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:35:06 (1788158106) [ 4986.191913] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4988.133521] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4990.066553] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:35:12 (1788158112) [ 4997.487301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.834371] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4997.875612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.090955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4998.100176] Lustre: Skipped 6 previous similar messages [ 5003.301862] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5012.460527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.547491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5012.582451] Lustre: 157340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5012.641109] LustreError: 157351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.657617] LustreError: 157351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5017.207192] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5023.701826] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5028.781322] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5034.713429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5034.927422] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5040.560878] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5043.878376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5046.852492] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5051.323765] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5051.554672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5055.779666] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5055.979811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5059.043756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5059.054092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.065461] Lustre: Skipped 5 previous similar messages [ 5063.899557] Lustre: server umount lustre-OST0000 complete [ 5085.628521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5092.706995] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5100.289505] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.338276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.355287] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5100.382958] Lustre: Skipped 1 previous similar message [ 5102.623920] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5109.008100] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5109.378470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5114.192680] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5114.531138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5121.224926] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9931c376f000.ost_server_uuid 50 [ 5122.688852] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9931c376f000.ost_server_uuid in FULL state after 0 sec [ 5123.103958] LustreError: 157347:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.24@tcp) returned error from blocking AST (req@00000000d6c245f4 x1875018707641856 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000729c8ecf/0x77f68d67d511d13e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.24@tcp remote: 0xafffbc6428e4e521 expref: 5 pid: 157346 timeout: 5223 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5123.128365] LustreError: lustre-MDT0000: A client on nid 192.168.203.24@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5123.135584] LustreError: 157336:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.24@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000729c8ecf/0x77f68d67d511d13e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.24@tcp remote: 0xafffbc6428e4e521 expref: 6 pid: 157346 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5129.192549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5129.209591] LustreError: Skipped 1 previous similar message [ 5132.435145] Lustre: server umount lustre-OST0000 complete [ 5142.821375] Lustre: server umount lustre-MDT0000 complete [ 5146.455431] LustreError: 157332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158271 with bad export cookie 8644252011880107139 [ 5146.471924] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.479054] LustreError: 157332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5152.974224] Lustre: server umount lustre-MDT0001 complete [ 5162.794053] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5164.974702] Key type lgssc unregistered [ 5165.191268] LNet: 163453:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5165.199074] LNetError: 163453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5165.217823] LNet: Removed LNI 192.168.203.124@tcp [ 5166.050431] Key type .llcrypt unregistered [ 5166.052722] Key type ._llcrypt unregistered [ 5185.832121] Key type ._llcrypt registered [ 5185.837120] Key type .llcrypt registered [ 5185.930685] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5186.617281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.692810] alg: No test for adler32 (adler32-zlib) [ 5187.753272] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5188.028504] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5189.752150] Key type lgssc registered [ 5190.729325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5198.961225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5199.008228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.548755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5204.451849] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5211.108738] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.126949] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5212.365560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.663754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.333199] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5222.792722] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5228.118825] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5234.296266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5234.687769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.429367] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5245.949490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5247.546446] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5254.273691] Lustre: Failing over lustre-MDT0000 [ 5254.499806] Lustre: server umount lustre-MDT0000 complete [ 5255.474561] LustreError: 164844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5256.164580] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.192106] Lustre: Skipped 2 previous similar messages [ 5261.282543] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.308765] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5266.401233] LustreError: 164847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.420938] LustreError: 164847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5266.632275] LustreError: 164827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.24@tcp arrived at 1788158391 with bad export cookie 16444527049027058524 [ 5271.525313] Lustre: 164158:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788158380/real 1788158380] req@000000005a54154f x1875019890188800/t0(0) o400->MGC192.168.203.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788158396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5271.526859] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.565661] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5271.597499] LustreError: 164848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5273.295532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.764790] LustreError: 164844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.766875] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe436b239688f7bbe [ 5281.787715] LustreError: 164844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5281.795732] Lustre: MGC192.168.203.124@tcp: Connection restored to 0@lo (at 0@lo) [ 5282.065750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5282.120839] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5286.399925] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5287.406523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5287.421176] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5288.428634] Lustre: lustre-MDT0000: Denying connection for new client f428be4c-0447-40ae-a6e3-8cf4a2e9803c (at 192.168.203.24@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5297.635602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.672565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5297.684862] Lustre: Skipped 1 previous similar message [ 5302.767352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5302.776170] Lustre: Skipped 1 previous similar message [ 5303.091765] Lustre: server umount lustre-OST0000 complete [ 5306.672292] LustreError: 168574:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5306.679730] Lustre: 167922:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5306.687513] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5306.708072] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5306.721907] LustreError: 167922:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fe9a842c x1875019890208896/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5306.737269] LustreError: 167922:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5306.744541] LustreError: 167922:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5306.757495] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5307.279752] Lustre: server umount lustre-MDT0000 complete [ 5307.873941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.895031] Lustre: Skipped 1 previous similar message [ 5307.899665] LustreError: 168286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5310.874888] LustreError: 164827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158435 with bad export cookie 16444527049027058622 [ 5310.892946] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5310.898061] LustreError: 164827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5313.263390] Lustre: server umount lustre-MDT0001 complete [ 5323.771194] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5325.990767] Key type lgssc unregistered [ 5326.275268] LNet: 169327:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5326.290939] LNetError: 169327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5326.313677] LNet: Removed LNI 192.168.203.124@tcp [ 5327.097194] Key type .llcrypt unregistered [ 5327.099250] Key type ._llcrypt unregistered [ 5349.383814] Key type ._llcrypt registered [ 5349.385588] Key type .llcrypt registered [ 5349.510957] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5350.467328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5350.511989] alg: No test for adler32 (adler32-zlib) [ 5351.549924] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5351.808568] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5353.536204] Key type lgssc registered [ 5354.762991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5361.900874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5361.922244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.356831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.412713] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5373.922873] LustreError: 170740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.970784] LustreError: 170740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5376.131176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5376.501266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5380.485553] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5388.726273] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5395.087836] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5402.900415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5403.394392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.520453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5411.163571] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5419.122068] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5427.169469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.171031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.177570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.187155] Lustre: Skipped 1 previous similar message [ 5431.434967] Lustre: server umount lustre-OST0000 complete [ 5437.410487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.422313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5437.427785] Lustre: Skipped 1 previous similar message [ 5441.662666] Lustre: server umount lustre-MDT0000 complete [ 5442.532118] LustreError: 172793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.583584] LustreError: 172793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5445.379172] LustreError: 170721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158570 with bad export cookie 8766225584277839989 [ 5445.386593] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5445.390750] LustreError: 170721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5445.631655] Lustre: server umount lustre-MDT0001 complete [ 5457.818473] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5460.851518] Key type lgssc unregistered [ 5461.118810] LNet: 174282:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5461.128589] LNetError: 174282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5462.185828] LNet: Removed LNI 192.168.203.124@tcp [ 5463.041753] Key type .llcrypt unregistered [ 5463.044299] Key type ._llcrypt unregistered [ 5492.729404] Key type ._llcrypt registered [ 5492.733971] Key type .llcrypt registered [ 5492.837786] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5493.872703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5493.946060] alg: No test for adler32 (adler32-zlib) [ 5494.965711] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5495.118368] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5496.752180] Key type lgssc registered [ 5497.716569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5504.714914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5504.726336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.188576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.491431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5515.756894] LustreError: 175985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.774537] LustreError: 175985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5518.347888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.802680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5523.216124] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5530.657389] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5535.906660] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5541.766135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.135465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.687651] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5551.353860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5555.201526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5561.834361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5561.836827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.843379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.891592] Lustre: Skipped 1 previous similar message [ 5566.950609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5566.960437] Lustre: Skipped 1 previous similar message [ 5567.413866] Lustre: server umount lustre-OST0000 complete [ 5572.066050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5572.090657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5572.103537] Lustre: Skipped 1 previous similar message [ 5577.186548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5577.196346] Lustre: Skipped 1 previous similar message [ 5582.306421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.315825] Lustre: Skipped 1 previous similar message [ 5585.376134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5585.559717] Lustre: server umount lustre-MDT0000 complete [ 5587.428686] LustreError: 175985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5587.447193] LustreError: 175985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5588.529666] LustreError: 175967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158713 with bad export cookie 14457133526927235451 [ 5588.532049] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5588.538255] LustreError: 175967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5588.829508] Lustre: server umount lustre-MDT0001 complete [ 5597.698659] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5599.996343] Key type lgssc unregistered [ 5600.223624] LNet: 179516:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5600.228439] LNetError: 179516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5600.248793] LNet: Removed LNI 192.168.203.124@tcp [ 5601.198302] Key type .llcrypt unregistered [ 5601.202391] Key type ._llcrypt unregistered [ 5621.654092] Key type ._llcrypt registered [ 5621.655768] Key type .llcrypt registered [ 5621.726881] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5622.538122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5622.577974] alg: No test for adler32 (adler32-zlib) [ 5623.592630] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5623.786816] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5625.432176] Key type lgssc registered [ 5626.592173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5636.679971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5636.699760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5638.137963] Lustre: *** cfs_fail_loc=135, val=0*** [ 5638.248846] LustreError: 181039:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5638.254486] LustreError: 181039:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.124@tcp: cfg command failed: rc = -2 [ 5638.277910] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5638.277910] [ 5638.293146] LustreError: MGC192.168.203.124@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5638.307398] LustreError: 181000:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5638.313641] LustreError: 181000:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 5638.325927] LustreError: 181000:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 5638.331089] LustreError: 181000:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5638.412271] Lustre: server umount lustre-MDT0000 complete [ 5638.417795] LustreError: 181000:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5645.161168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.722645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.065030] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5656.035738] LustreError: 181410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.069775] LustreError: 181410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5658.012541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5658.400977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5663.311305] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5670.908591] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5676.134814] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5677.937478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.942224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5677.956349] Lustre: Skipped 1 previous similar message [ 5683.106176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.115018] Lustre: Skipped 2 previous similar messages [ 5688.292763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5688.296575] Lustre: Skipped 1 previous similar message [ 5692.384156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5692.621556] Lustre: server umount lustre-MDT0000 complete [ 5693.416340] LustreError: 181405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.444467] LustreError: 181405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5696.366234] LustreError: 181391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158821 with bad export cookie 13030858957023430002 [ 5696.377458] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5696.382859] LustreError: 181391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5696.766603] Lustre: server umount lustre-MDT0001 complete [ 5707.291817] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5709.658850] Key type lgssc unregistered [ 5709.895241] LNet: 183730:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5709.902732] LNetError: 183730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5709.917428] LNet: Removed LNI 192.168.203.124@tcp [ 5710.726716] Key type .llcrypt unregistered [ 5710.728520] Key type ._llcrypt unregistered [ 5731.437602] Key type ._llcrypt registered [ 5731.440572] Key type .llcrypt registered [ 5731.516654] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5731.760286] Key type .llcrypt unregistered [ 5731.763048] Key type ._llcrypt unregistered [ 5747.987433] Key type ._llcrypt registered [ 5747.989608] Key type .llcrypt registered [ 5748.067427] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5748.889053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5748.936794] alg: No test for adler32 (adler32-zlib) [ 5750.103964] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5750.335287] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5752.032222] Key type lgssc registered [ 5752.993204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5759.656881] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5759.694684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5775.200986] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5780.320334] LustreError: 185864:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 5807.072298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.988109] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5822.496368] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5829.703529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.146989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.206072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5834.732444] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5835.239539] LustreError: 185890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5835.255527] LustreError: 185890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5840.355387] LustreError: 185889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.408294] LustreError: 187556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.439708] LustreError: 187556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5843.856651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.435480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.143417] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5857.605828] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5863.359932] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5865.847253] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5876.197192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5876.206198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.219833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5880.292278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5880.295996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.311271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5880.317962] Lustre: Skipped 1 previous similar message [ 5882.127884] Lustre: server umount lustre-OST0000 complete [ 5890.529439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5890.534394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.538182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5890.544362] Lustre: Skipped 1 previous similar message [ 5891.972521] Lustre: server umount lustre-MDT0000 complete [ 5895.652241] LustreError: 186906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.689634] LustreError: 186906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5896.460541] LustreError: 188525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159021 with bad export cookie 10540897842616566221 [ 5896.483918] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5896.491941] LustreError: 188525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5896.953104] Lustre: server umount lustre-MDT0001 complete [ 5908.006164] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5911.898653] Key type lgssc unregistered [ 5912.284713] LNet: 189829:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5912.296869] LNetError: 189829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5912.328120] LNet: Removed LNI 192.168.203.124@tcp [ 5913.524742] Key type .llcrypt unregistered [ 5913.527407] Key type ._llcrypt unregistered [ 5936.542898] Key type ._llcrypt registered [ 5936.547391] Key type .llcrypt registered [ 5936.670671] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5937.689933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5937.767580] alg: No test for adler32 (adler32-zlib) [ 5938.804332] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5939.011399] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5940.648837] Key type lgssc registered [ 5941.649771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5949.914679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5949.934496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.281896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.060765] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5960.686427] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.712188] LustreError: 191240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5964.514786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.814650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5970.672389] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5979.948391] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5986.479516] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5993.035847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5993.443604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.547461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6000.886195] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6008.831890] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6014.201823] Lustre: Failing over lustre-MDT0000 [ 6014.416461] Lustre: server umount lustre-MDT0000 complete [ 6015.977317] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.981424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6016.005026] LustreError: 192530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6016.033352] LustreError: 192530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6021.089587] LustreError: 191235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6021.089587] LustreError: 191236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6021.089599] LustreError: 191236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6021.121162] LustreError: 191235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6026.209799] LustreError: 191968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.220846] LustreError: 191968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6031.329309] Lustre: 190552:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788159140/real 1788159140] req@0000000010d11eef x1875020677670144/t0(0) o400->MGC192.168.203.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788159156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6031.384772] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6033.760389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.451994] LustreError: 192530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.473199] LustreError: 192530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6041.574145] LustreError: 190551:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bb920dc3 x1875020677675392/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6041.831678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.887769] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6042.364520] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6046.116307] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6047.252630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6047.283658] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6047.342794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6052.236864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6064.278169] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6072.305539] Lustre: server umount lustre-OST0000 complete [ 6077.933164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.940760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6077.941877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6077.950713] Lustre: Skipped 3 previous similar messages [ 6077.966496] Lustre: Skipped 1 previous similar message [ 6083.043676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.055219] Lustre: Skipped 1 previous similar message [ 6086.249562] Lustre: server umount lustre-MDT0000 complete [ 6088.161946] Lustre: 190554:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788159196/real 1788159196] req@000000005a54154f x1875020677700864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788159212 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6088.179385] LustreError: 191237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.190922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.232022] LustreError: 191237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6090.676155] LustreError: 191223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159215 with bad export cookie 8656123494928635774 [ 6090.682393] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.710402] LustreError: 191223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6091.063783] Lustre: server umount lustre-MDT0001 complete [ 6104.826667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6107.418654] Key type lgssc unregistered [ 6107.642936] LNet: 196226:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.649808] LNetError: 196226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.662950] LNet: Removed LNI 192.168.203.124@tcp [ 6108.488366] Key type .llcrypt unregistered [ 6108.490726] Key type ._llcrypt unregistered [ 6131.419743] Key type ._llcrypt registered [ 6131.421734] Key type .llcrypt registered [ 6131.542696] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6132.317450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6132.339732] alg: No test for adler32 (adler32-zlib) [ 6133.415674] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6133.614891] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6135.336920] Key type lgssc registered [ 6136.440342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.068214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.083545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.449804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.846443] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6155.757404] LustreError: 197639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.786090] LustreError: 197639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6156.682481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.012465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6161.505811] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6168.604251] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6174.552808] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6181.108719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6181.446135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6187.230949] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6190.706302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6194.574232] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6197.660761] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6203.193306] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6217.118738] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6223.842847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.863916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.879754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.918077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.921076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.949251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.816379] Lustre: server umount lustre-OST0000 complete [ 6234.081321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6234.090253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.100760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6234.108581] Lustre: Skipped 1 previous similar message [ 6237.153411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6237.158279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.799042] Lustre: server umount lustre-MDT0000 complete [ 6242.275002] LustreError: 197639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.293535] LustreError: 197639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6242.344334] LustreError: 197620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159367 with bad export cookie 3873713510376307883 [ 6242.354194] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.365083] LustreError: 197620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6242.605947] Lustre: server umount lustre-MDT0001 complete [ 6253.891721] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6256.761944] Key type lgssc unregistered [ 6257.044165] LNet: 201481:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6257.055791] LNetError: 201481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6257.084953] LNet: Removed LNI 192.168.203.124@tcp [ 6257.763762] Key type .llcrypt unregistered [ 6257.768652] Key type ._llcrypt unregistered [ 6276.025454] Key type ._llcrypt registered [ 6276.028931] Key type .llcrypt registered [ 6276.133871] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6277.382790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6277.417775] alg: No test for adler32 (adler32-zlib) [ 6278.579173] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6278.864499] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6280.584205] Key type lgssc registered [ 6281.921991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6289.479068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6289.512307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.056679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6296.096207] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6300.657842] LustreError: 202749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.683228] LustreError: 202749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6305.150346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.615870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6310.044664] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6317.512647] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6322.782409] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6330.147260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6330.352944] LustreError: 204569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.611535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.760322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6338.481688] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6346.977936] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6352.198423] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6363.764643] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6371.524282] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6376.336128] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6387.169073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6387.172755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.181606] LustreError: Skipped 1 previous similar message [ 6387.182966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.228898] Lustre: Skipped 1 previous similar message [ 6389.687375] Lustre: server umount lustre-OST0000 complete [ 6397.410304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.439515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6397.451103] Lustre: Skipped 1 previous similar message [ 6399.507234] Lustre: server umount lustre-MDT0000 complete [ 6402.529135] LustreError: 202749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6402.541963] LustreError: 202749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6403.258045] LustreError: 202732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159527 with bad export cookie 9496757778113752099 [ 6403.261292] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6403.273125] LustreError: 202732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6403.580498] Lustre: server umount lustre-MDT0001 complete [ 6415.438639] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6418.385479] Key type lgssc unregistered [ 6418.718706] LNet: 207579:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6418.724967] LNetError: 207579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6418.744435] LNet: Removed LNI 192.168.203.124@tcp [ 6419.718458] Key type .llcrypt unregistered [ 6419.720316] Key type ._llcrypt unregistered [ 6445.305726] Key type ._llcrypt registered [ 6445.308651] Key type .llcrypt registered [ 6445.391934] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6446.449258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6446.558759] alg: No test for adler32 (adler32-zlib) [ 6447.583284] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6447.809677] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6449.513131] Key type lgssc registered [ 6450.530190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6457.560114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6457.576116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.131150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6463.141928] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6468.591063] LustreError: 208994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6468.624464] LustreError: 208994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6471.246104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.530687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6475.260883] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6481.765635] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6486.818076] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.714519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6493.161445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.551794] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6503.402982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6504.707962] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6517.729165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6517.740262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.761622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.757780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.759042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.761190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.781272] Lustre: Skipped 1 previous similar message [ 6522.726699] Lustre: server umount lustre-OST0000 complete [ 6527.971569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6527.982666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.005904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.114103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.121490] Lustre: Skipped 3 previous similar messages [ 6539.239420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6539.244674] Lustre: Skipped 1 previous similar message [ 6540.256690] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6540.440222] Lustre: server umount lustre-MDT0000 complete [ 6543.349057] LustreError: 211436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.362844] LustreError: 211436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6544.090867] LustreError: 208973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159668 with bad export cookie 261479174962200078 [ 6544.096832] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.101243] LustreError: 208973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6544.644227] Lustre: server umount lustre-MDT0001 complete [ 6557.189997] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6560.563896] Key type lgssc unregistered [ 6560.895476] LNet: 212688:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6560.911482] LNetError: 212688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6561.961416] LNet: Removed LNI 192.168.203.124@tcp [ 6563.271196] Key type .llcrypt unregistered [ 6563.276763] Key type ._llcrypt unregistered [ 6581.643713] Key type ._llcrypt registered [ 6581.648845] Key type .llcrypt registered [ 6581.794552] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6582.546839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6582.627039] alg: No test for adler32 (adler32-zlib) [ 6583.727585] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6583.962134] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6585.616238] Key type lgssc registered [ 6586.606400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6594.210589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6594.225506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.799715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.948079] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6606.308401] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.335811] LustreError: 213947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6609.193920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.857535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6614.153946] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6621.341373] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6627.133692] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6636.205552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.104337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.210125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6645.249020] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6652.358622] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6667.745294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6667.751832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.768688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.797346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.800979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.801789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.830551] Lustre: Skipped 1 previous similar message [ 6671.462757] Lustre: server umount lustre-OST0000 complete [ 6676.449336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6676.457184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.473377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6680.035656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6680.041791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.399256] Lustre: server umount lustre-MDT0000 complete [ 6684.903719] LustreError: 213927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159809 with bad export cookie 2082143457101549853 [ 6684.907479] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6684.919273] LustreError: 213927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6685.231196] Lustre: server umount lustre-MDT0001 complete [ 6695.084221] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6697.802163] Key type lgssc unregistered [ 6698.086637] LNet: 217633:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6698.105219] LNetError: 217633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6698.119564] LNet: Removed LNI 192.168.203.124@tcp [ 6698.952478] Key type .llcrypt unregistered [ 6698.956392] Key type ._llcrypt unregistered [ 6715.441841] Key type ._llcrypt registered [ 6715.444509] Key type .llcrypt registered [ 6715.546394] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6716.301126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6716.366736] alg: No test for adler32 (adler32-zlib) [ 6717.414744] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6717.648964] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6719.296190] Key type lgssc registered [ 6720.605465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6728.206747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6728.226255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.682245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.080310] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6739.939237] LustreError: 218880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6739.957666] LustreError: 218880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6742.058931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6742.462304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.481577] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6754.189231] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6759.552816] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6765.691993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6766.141950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.474840] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6775.360930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6780.143306] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.006962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6788.068342] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6788.309953] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6793.407614] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6800.362634] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6801.785403] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6813.582051] Lustre: Permanently deactivating lustre-OST0001 [ 6813.603710] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6823.450501] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6831.357581] Lustre: Permanently reactivating lustre-OST0001 [ 6831.370640] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6831.375481] Lustre: Skipped 2 previous similar messages [ 6837.814336] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.832678] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6837.839870] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6837.846850] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6837.866592] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6841.622672] Lustre: server umount lustre-OST0001 complete [ 6842.878531] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.903258] Lustre: Skipped 1 previous similar message [ 6842.911740] LustreError: 221543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.933669] LustreError: 221543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6846.948727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6846.958951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.981303] Lustre: Skipped 1 previous similar message [ 6846.986919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.977899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.986983] LustreError: 220701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6848.019804] LustreError: 220701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6853.098872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6853.101802] LustreError: 220701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6853.111153] Lustre: Skipped 2 previous similar messages [ 6853.129829] LustreError: 220701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6858.220915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6858.222841] LustreError: 220699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.234396] Lustre: Skipped 1 previous similar message [ 6858.273874] LustreError: 220699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6859.232537] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6859.358566] Lustre: server umount lustre-OST0000 complete [ 6865.387628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6865.405813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6865.429789] Lustre: Skipped 1 previous similar message [ 6865.447384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6869.785165] Lustre: server umount lustre-MDT0000 complete [ 6873.571965] LustreError: 218880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6873.602199] LustreError: 218880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6873.713955] LustreError: 222714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159998 with bad export cookie 9242437195133775502 [ 6873.720617] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6873.730629] LustreError: 222714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6874.064699] Lustre: server umount lustre-MDT0001 complete [ 6885.709824] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6888.330241] Key type lgssc unregistered [ 6888.584736] LNet: 223979:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6888.589569] LNetError: 223979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6888.603777] LNet: Removed LNI 192.168.203.124@tcp [ 6889.416491] Key type .llcrypt unregistered [ 6889.418840] Key type ._llcrypt unregistered [ 6927.465563] Key type ._llcrypt registered [ 6927.469505] Key type .llcrypt registered [ 6927.599948] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6928.742019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6928.775916] alg: No test for adler32 (adler32-zlib) [ 6929.863225] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6930.158492] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6931.856464] Key type lgssc registered [ 6932.830789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6939.994200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6940.018173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.214847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6941.228890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6941.333320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.582398] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6953.821880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.902938] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6953.926262] Lustre: 226181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6954.213706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6959.011975] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6966.494855] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6972.247522] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6978.893229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6979.020881] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6979.211210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.063694] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6986.399447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6992.388932] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6995.034377] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7001.503085] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7006.919704] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7016.746829] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7025.600977] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7043.322690] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7043.330601] Lustre: Skipped 1 previous similar message [ 7082.146719] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7082.155864] Lustre: Skipped 4 previous similar messages [ 7146.791747] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7146.807750] Lustre: Skipped 7 previous similar messages [ 7175.652212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7175.661687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7175.679686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.164139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7176.165188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.204845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7176.207511] Lustre: Skipped 1 previous similar message [ 7177.655356] Lustre: server umount lustre-OST0000 complete [ 7181.281598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.302523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7181.307640] Lustre: Skipped 1 previous similar message [ 7186.402835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7186.411100] Lustre: Skipped 1 previous similar message [ 7187.372324] Lustre: server umount lustre-MDT0000 complete [ 7190.551624] LustreError: 228749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160315 with bad export cookie 8417998817376791862 [ 7190.559274] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7190.562344] LustreError: 228749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7190.825506] Lustre: server umount lustre-MDT0001 complete [ 7200.868955] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7203.138832] Key type lgssc unregistered [ 7203.389981] LNet: 230954:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7203.397261] LNetError: 230954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7204.457115] LNet: Removed LNI 192.168.203.124@tcp [ 7205.216831] Key type .llcrypt unregistered [ 7205.220099] Key type ._llcrypt unregistered [ 7226.033304] Key type ._llcrypt registered [ 7226.036549] Key type .llcrypt registered [ 7226.159810] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7227.183788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7227.214735] alg: No test for adler32 (adler32-zlib) [ 7228.217171] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7228.382810] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7230.024216] Key type lgssc registered [ 7231.181241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7238.298536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7238.325835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7239.783077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.081901] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7249.900203] LustreError: 232347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.915039] LustreError: 232347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7251.432716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.856979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7255.730343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7262.813675] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7268.633302] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7275.206167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7275.525800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.456556] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 7284.740883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7288.642468] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7302.117336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7302.127054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.150509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7305.187946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.189468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7305.193878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7308.337374] Lustre: server umount lustre-OST0000 complete [ 7313.377519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7313.393516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.412437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7313.420236] Lustre: Skipped 1 previous similar message [ 7315.425647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7315.436840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.551565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7320.561274] Lustre: Skipped 2 previous similar messages [ 7326.176661] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7326.387623] Lustre: server umount lustre-MDT0000 complete [ 7329.634883] LustreError: 232328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160454 with bad export cookie 6434606576620105199 [ 7329.636452] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7329.650216] LustreError: 232328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7329.937997] Lustre: server umount lustre-MDT0001 complete [ 7339.615053] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7342.566983] Key type lgssc unregistered [ 7342.846592] LNet: 236041:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7342.863170] LNetError: 236041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7342.902227] LNet: Removed LNI 192.168.203.124@tcp [ 7343.753957] Key type .llcrypt unregistered [ 7343.758678] Key type ._llcrypt unregistered [ 7366.043370] Key type ._llcrypt registered [ 7366.045936] Key type .llcrypt registered [ 7366.115498] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7366.894614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7367.029816] alg: No test for adler32 (adler32-zlib) [ 7368.017530] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7368.202609] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7369.825452] Key type lgssc registered [ 7370.925993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7640.837251] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7644.528486] Key type lgssc unregistered [ 7644.855707] LNet: 238065:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7644.863611] LNetError: 238065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7644.886199] LNet: Removed LNI 192.168.203.124@tcp [ 7645.951697] Key type .llcrypt unregistered [ 7645.956358] Key type ._llcrypt unregistered [ 7673.762125] Key type ._llcrypt registered [ 7673.773644] Key type .llcrypt registered [ 7673.911488] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 7689.285678] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 7690.307429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7690.389991] alg: No test for adler32 (adler32-zlib) [ 7691.692221] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7691.979569] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 7693.792326] Key type lgssc registered [ 7694.962734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.029548] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7471 sec ========== 03:20:51 (1788160851) [ 7730.787703] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:20:53 (1788160853) === [ 7734.217852] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:20:56 (1788160856) === [ 7773.554867] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7776.133319] Key type lgssc unregistered [ 7776.452662] LNet: 243301:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7776.460413] LNetError: 243301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7776.473581] LNet: Removed LNI 192.168.203.124@tcp [ 7777.220503] Key type .llcrypt unregistered [ 7777.223529] Key type ._llcrypt unregistered