[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 526439516 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002344] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006601] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009114] LSM: Security Framework initializing [ 0.010036] Yama: becoming mindful. [ 0.011024] SELinux: Initializing. [ 0.013018] *** VALIDATE selinux *** [ 0.020232] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024707] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027080] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028091] *** VALIDATE tmpfs *** [ 0.030324] *** VALIDATE proc *** [ 0.031182] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.034159] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035140] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037021] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041374] debug: unmapping init [mem 0xffffffffb6459000-0xffffffffb6460fff] [ 0.043143] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044550] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049008] ... max period: 00007fffffffffff [ 0.050009] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052246] rcu: Hierarchical SRCU implementation. [ 0.054269] smp: Bringing up secondary CPUs ... [ 0.055470] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.059158] smp: Brought up 1 node, 4 CPUs [ 0.061010] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141025] node 0 deferred pages initialised in 76ms [ 0.146008] devtmpfs: initialized [ 0.147263] x86/mm: Memory block size: 128MB [ 0.150814] gcov: version magic: 0x41383552 [ 0.154253] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160240] pinctrl core: initialized pinctrl subsystem [ 0.162114] [ 0.162684] ************************************************************* [ 0.165008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167007] ** ** [ 0.169008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172007] ** ** [ 0.174007] ** This means that this kernel is built to expose internal ** [ 0.176010] ** IOMMU data structures, which may compromise security on ** [ 0.179008] ** your system. ** [ 0.181006] ** ** [ 0.183008] ** If you see this message and you are not debugging the ** [ 0.186008] ** kernel, report this immediately to your vendor! ** [ 0.188006] ** ** [ 0.190008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193008] ************************************************************* [ 0.195730] NET: Registered protocol family 16 [ 0.198358] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207039] cpuidle: using governor menu [ 0.209019] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212556] PCI: Using configuration type 1 for base access [ 0.214115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226034] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227009] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230025] cryptd: max_cpu_qlen set to 1000 [ 0.232174] ACPI: Added _OSI(Module Device) [ 0.234010] ACPI: Added _OSI(Processor Device) [ 0.235009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242135] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247480] ACPI: Interpreter enabled [ 0.249038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251009] ACPI: Using IOAPIC for interrupt routing [ 0.252092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256346] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265640] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282292] acpiphp: Slot [2] registered [ 0.284102] acpiphp: Slot [5] registered [ 0.286079] acpiphp: Slot [6] registered [ 0.287103] acpiphp: Slot [7] registered [ 0.289090] acpiphp: Slot [8] registered [ 0.291064] acpiphp: Slot [9] registered [ 0.293106] acpiphp: Slot [10] registered [ 0.294080] acpiphp: Slot [3] registered [ 0.296094] acpiphp: Slot [4] registered [ 0.297090] acpiphp: Slot [11] registered [ 0.299093] acpiphp: Slot [12] registered [ 0.301130] acpiphp: Slot [13] registered [ 0.302073] acpiphp: Slot [14] registered [ 0.304077] acpiphp: Slot [15] registered [ 0.306171] acpiphp: Slot [16] registered [ 0.307092] acpiphp: Slot [17] registered [ 0.309072] acpiphp: Slot [18] registered [ 0.310064] acpiphp: Slot [19] registered [ 0.312072] acpiphp: Slot [20] registered [ 0.313097] acpiphp: Slot [21] registered [ 0.315138] acpiphp: Slot [22] registered [ 0.317089] acpiphp: Slot [23] registered [ 0.319085] acpiphp: Slot [24] registered [ 0.321135] acpiphp: Slot [25] registered [ 0.323056] acpiphp: Slot [26] registered [ 0.324086] acpiphp: Slot [27] registered [ 0.326088] acpiphp: Slot [28] registered [ 0.328079] acpiphp: Slot [29] registered [ 0.330085] acpiphp: Slot [30] registered [ 0.332106] acpiphp: Slot [31] registered [ 0.334050] PCI host bridge to bus 0000:00 [ 0.335016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.349019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356543] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360372] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370780] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.376033] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.384014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388496] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.392196] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.398836] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.405013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.418014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.433216] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.462012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.494436] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.509012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536126] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.565019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.605018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.622599] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.629013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.636013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.650019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.660012] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.667014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.672013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.692016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.700487] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.718012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.733012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.783012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.807948] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.809297] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.812294] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.814346] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.816168] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.822030] iommu: Default domain type: Passthrough [ 0.823363] SCSI subsystem initialized [ 0.825101] ACPI: bus type USB registered [ 0.826104] usbcore: registered new interface driver usbfs [ 0.828046] usbcore: registered new interface driver hub [ 0.830060] usbcore: registered new device driver usb [ 0.832158] pps_core: LinuxPPS API ver. 1 registered [ 0.833007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.837055] PTP clock support registered [ 0.839036] EDAC MC: Ver: 3.0.0 [ 0.841107] PCI: Using ACPI for IRQ routing [ 0.842731] NetLabel: Initializing [ 0.844010] NetLabel: domain hash size = 128 [ 0.846008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.848070] NetLabel: unlabeled traffic allowed by default [ 0.850138] vgaarb: loaded [ 0.853027] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.854009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.862000] clocksource: Switched to clocksource kvm-clock [ 0.964375] VFS: Disk quotas dquot_6.6.0 [ 0.966030] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.968578] *** VALIDATE ramfs *** [ 0.969884] *** VALIDATE hugetlbfs *** [ 0.971619] pnp: PnP ACPI init [ 0.973985] pnp: PnP ACPI: found 6 devices [ 1.017800] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.021261] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.023450] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.025595] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.028108] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.030561] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.033600] NET: Registered protocol family 2 [ 1.036506] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.044777] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.048480] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.054120] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.058047] TCP: Hash tables configured (established 65536 bind 65536) [ 1.061354] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.065084] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.068228] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.071746] NET: Registered protocol family 1 [ 1.074847] RPC: Registered named UNIX socket transport module. [ 1.077201] RPC: Registered udp transport module. [ 1.078862] RPC: Registered tcp transport module. [ 1.080650] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.083274] NET: Registered protocol family 44 [ 1.084893] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.087134] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.089222] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.091848] PCI: CLS 0 bytes, default 64 [ 1.093600] Unpacking initramfs... [ 2.455466] debug: unmapping init [mem 0xffff8f033cc54000-0xffff8f033ffbffff] [ 2.459366] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.462848] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.465297] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.972185] Initialise system trusted keyrings [ 2.975262] Key type blacklist registered [ 2.978108] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.987771] zbud: loaded [ 2.991155] *** VALIDATE nfs *** [ 2.992364] *** VALIDATE nfs4 *** [ 2.993714] pstore: using deflate compression [ 3.001240] Platform Keyring initialized [ 3.113725] NET: Registered protocol family 38 [ 3.116095] Key type asymmetric registered [ 3.118243] Asymmetric key parser 'x509' registered [ 3.120672] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.125130] io scheduler mq-deadline registered [ 3.127350] io scheduler kyber registered [ 3.129479] io scheduler bfq registered [ 3.131723] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.135524] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.138894] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.142980] ACPI: Power Button [PWRF] [ 3.148531] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.156073] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.174640] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.201118] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.242757] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.271993] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.300494] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.306267] Non-volatile memory driver v1.3 [ 3.308271] Linux agpgart interface v0.103 [ 3.333820] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.337312] vda: detected capacity change from 0 to 74694656 [ 3.363636] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.367166] vdb: detected capacity change from 0 to 1073741824 [ 3.388732] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.392078] vdc: detected capacity change from 0 to 2621440000 [ 3.423076] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.426648] vdd: detected capacity change from 0 to 2621440000 [ 3.454669] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.457352] vde: detected capacity change from 0 to 4294967296 [ 3.484463] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.487813] vdf: detected capacity change from 0 to 4294967296 [ 3.508202] libphy: Fixed MDIO Bus: probed [ 3.516238] usbcore: registered new interface driver usbserial_generic [ 3.518798] usbserial: USB Serial support registered for generic [ 3.521137] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.525541] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.527696] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.530788] mousedev: PS/2 mouse device common for all mice [ 3.533836] rtc_cmos 00:05: RTC can wake from S4 [ 3.537753] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.541379] rtc_cmos 00:05: registered as rtc0 [ 3.543238] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.546123] intel_pstate: CPU model not supported [ 3.549411] hid: raw HID events driver (C) Jiri Kosina [ 3.551807] usbcore: registered new interface driver usbhid [ 3.553750] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.554707] usbhid: USB HID core driver [ 3.559205] drop_monitor: Initializing network drop monitor service [ 3.561962] Initializing XFRM netlink socket [ 3.563849] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.564123] NET: Registered protocol family 10 [ 3.569624] Segment Routing with IPv6 [ 3.570952] NET: Registered protocol family 17 [ 3.573276] mpls_gso: MPLS GSO support [ 3.579172] RAS: Correctable Errors collector initialized. [ 3.581627] AVX version of gcm_enc/dec engaged. [ 3.583675] AES CTR mode by8 optimization enabled [ 3.658390] sched_clock: Marking stable (3658360237, 0)->(4475383969, -817023732) [ 3.662391] registered taskstats version 1 [ 3.664744] Loading compiled-in X.509 certificates [ 3.667337] zswap: loaded using pool lzo/zbud [ 3.698057] Key type big_key registered [ 3.710494] Key type encrypted registered [ 3.712352] ima: No TPM chip found, activating TPM-bypass! [ 3.714728] ima: Allocated hash algorithm: sha1 [ 3.716570] ima: No architecture policies found [ 3.719655] evm: Initialising EVM extended attributes: [ 3.721721] evm: security.selinux [ 3.723270] evm: security.ima [ 3.724676] evm: security.capability [ 3.726302] evm: HMAC attrs: 0x1 [ 3.729191] rtc_cmos 00:05: setting system clock to 2026-08-05 12:16:03 UTC (1785932163) [ 3.736963] debug: unmapping init [mem 0xffffffffb7403000-0xffffffffb75fffff] [ 3.741141] debug: unmapping init [mem 0xffffffffb6182000-0xffffffffb6458fff] [ 3.750083] Write protecting the kernel read-only data: 28672k [ 3.753647] debug: unmapping init [mem 0xffffffffb4803000-0xffffffffb49fffff] [ 3.756232] debug: unmapping init [mem 0xffffffffb5114000-0xffffffffb51fffff] [ 3.798744] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.806506] systemd[1]: Detected virtualization kvm. [ 3.808498] systemd[1]: Detected architecture x86-64. [ 3.810250] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.833614] systemd[1]: No hostname configured. [ 3.835249] systemd[1]: Set hostname to . [ 3.837067] random: systemd: uninitialized urandom read (16 bytes read) [ 3.839371] systemd[1]: Initializing machine ID from random generator. [ 3.901782] random: ln: uninitialized urandom read (6 bytes read) [ 4.004249] random: systemd: uninitialized urandom read (16 bytes read) [ 4.006826] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.010618] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.014769] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.632783] device-mapper: uevent: version 1.0.3 [ 4.635098] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.391976] random: fast init done [ 5.404200] virtio_net virtio0 ens2: renamed from eth0 [ 5.417434] scsi host0: ata_piix [ 5.441697] scsi host1: ata_piix [ 5.443985] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.449776] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.892517] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.232070] random: crng init done [ 10.233627] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.671883] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.753252] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.998326] SELinux: Disabled at runtime. [ 12.060894] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.067118] systemd[1]: Detected virtualization kvm. [ 12.068528] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.547527] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.551220] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.556042] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.562273] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.566341] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.573311] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.580811] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File S[ 12.663960] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ystems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.005110] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.365179] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.425229] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.554813] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.568757] EDAC sbridge: Ver: 1.1.2 [ 14.832486] Key type dns_resolver registered [ 15.117026] NFS: Registering the id_resolver key type [ 15.118887] Key type id_resolver registered [ 15.120345] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg321-server login: [ 39.878939] libcfs: loading out-of-tree module taints kernel. [ 39.900961] Key type ._llcrypt registered [ 39.902464] Key type .llcrypt registered [ 39.969511] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 71.772735] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 73.822446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.843062] alg: No test for adler32 (adler32-zlib) [ 75.554739] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 76.072023] hrtimer: interrupt took 2275862 ns [ 77.373508] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 79.487670] Key type lgssc registered [ 82.633731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.823810] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.934453] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 180.396656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.462111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.913559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.950783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.187708] Lustre: lustre-MDT0000: new disk, initializing [ 182.331951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.367130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.394443] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.045807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.168837] Lustre: 6534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.216584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.224641] Lustre: Skipped 1 previous similar message [ 210.344692] Lustre: lustre-MDT0001: new disk, initializing [ 210.462826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.507960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.530502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.553879] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.872350] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.675468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.983226] Lustre: lustre-OST0000: new disk, initializing [ 234.993148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.007348] Lustre: 8472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.082811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.681038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.697057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.813369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 243.638579] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.264834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.725845] Lustre: lustre-OST0001: new disk, initializing [ 264.747136] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.770953] Lustre: 9546:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.954868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.934884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 271.973770] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.092315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.674769] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.456058] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.651689] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 314.629928] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing check_logdir /tmp/testlogs/ [ 322.705841] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing yml_node [ 328.531658] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 332.292294] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 335.789764] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 337.864771] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 5 08:21:35 EDT 2026 [ 361.554624] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 364.384786] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 384.479748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 384.485620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.507640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.536105] Lustre: Skipped 1 previous similar message [ 384.568426] Lustre: Skipped 3 previous similar messages [ 388.941694] Lustre: server umount lustre-MDT0000 complete [ 393.561945] LustreError: 7482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785932553 with bad export cookie 9253794886199795049 [ 393.570027] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.576041] LustreError: 7482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 394.722329] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.724277] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.733542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 394.742072] Lustre: Skipped 3 previous similar messages [ 394.814826] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 398.755285] LustreError: 6541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.755922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 398.805887] Lustre: Skipped 2 previous similar messages [ 400.115140] Lustre: server umount lustre-MDT0001 complete [ 405.426283] Lustre: server umount lustre-OST0000 complete [ 409.879283] Lustre: server umount lustre-OST0001 complete [ 418.356093] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 435.491171] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 481.775909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.209279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 482.263364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 482.381457] Lustre: lustre-MDT0000: new disk, initializing [ 482.540363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.587818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 488.546234] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 501.529455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.746186] Lustre: 15128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 501.940702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 501.954223] Lustre: Skipped 1 previous similar message [ 502.142850] Lustre: lustre-MDT0001: new disk, initializing [ 502.363106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 502.439266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 502.457217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 509.796618] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 525.190942] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.997701] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 542.231637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 542.752781] Lustre: lustre-OST0000: new disk, initializing [ 542.758822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 542.767530] Lustre: 17100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 542.911957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.285315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 544.297371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 544.456066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 550.509544] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 563.772977] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.728157] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 570.923870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 577.481476] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 577.897721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 585.709464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.729335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 585.749795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 585.764667] LustreError: Skipped 1 previous similar message [ 585.793579] Lustre: Skipped 1 previous similar message [ 587.453322] Lustre: server umount lustre-OST0000 complete [ 594.406643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 594.472706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.480379] Lustre: Skipped 1 previous similar message [ 594.484558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 599.882084] Lustre: server umount lustre-MDT0000 complete [ 601.059866] LustreError: 15135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.093214] LustreError: 15135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 606.136325] LustreError: 18440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785932765 with bad export cookie 9253794886199796575 [ 606.145861] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.155189] LustreError: 18440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 606.991403] Lustre: server umount lustre-MDT0001 complete [ 613.200925] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:26:10 (1785932770) [ 622.317682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.158497] LustreError: 19369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.292291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.709460] LustreError: 19370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.054799] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 637.926324] LustreError: 19369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.964957] LustreError: 19369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 642.183566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.198508] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 659.511660] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 669.101165] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 684.275233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 694.398279] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 703.183525] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 717.015234] Lustre: server umount lustre-OST0000 complete [ 723.936789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.944129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 723.952672] Lustre: Skipped 2 previous similar messages [ 723.966591] Lustre: Skipped 3 previous similar messages [ 735.199148] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785932878/real 1785932878] req@ffff8f039dd97800 x1872685681545600/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785932894 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 735.202315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.247219] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 736.226296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 736.615480] Lustre: server umount lustre-MDT0000 complete [ 739.487175] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785932883/real 1785932883] req@ffff8f0287c3f800 x1872685681547392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785932899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 740.325159] LustreError: 19370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.360948] LustreError: 19370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 741.594706] LustreError: 19351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785932901 with bad export cookie 9253794886199797464 [ 741.619146] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.296724] Lustre: server umount lustre-MDT0001 complete [ 758.119390] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 762.193942] Key type lgssc unregistered [ 762.602777] LNet: 22912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.616460] LNetError: 22912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.651619] LNet: Removed LNI 192.168.203.121@tcp [ 764.174125] Key type .llcrypt unregistered [ 764.178431] Key type ._llcrypt unregistered [ 791.572614] Key type ._llcrypt registered [ 791.579100] Key type .llcrypt registered [ 791.791236] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 793.597513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 793.725066] alg: No test for adler32 (adler32-zlib) [ 795.060148] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 795.483224] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 797.319382] Key type lgssc registered [ 799.376114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 808.973900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.987776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.525572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.957318] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 820.708460] LustreError: 24338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.723096] LustreError: 24338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 824.449797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.026513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 830.801545] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 840.533133] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 848.625881] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 857.368651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.849436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.297735] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 877.044396] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.393712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 901.093014] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.112202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 901.118996] Lustre: Skipped 1 previous similar message [ 901.136894] Lustre: Skipped 1 previous similar message [ 906.210778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.223116] Lustre: Skipped 1 previous similar message [ 911.333137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.346712] Lustre: Skipped 1 previous similar message [ 912.863122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 913.286166] Lustre: server umount lustre-OST0000 complete [ 921.568449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.575925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.581974] Lustre: Skipped 1 previous similar message [ 926.692660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.706564] Lustre: Skipped 2 previous similar messages [ 932.319121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 932.886690] Lustre: server umount lustre-MDT0000 complete [ 936.930422] LustreError: 24339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.954157] LustreError: 24339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 936.983458] LustreError: 24319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785933096 with bad export cookie 16620312418028591053 [ 936.991924] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.996802] LustreError: 24319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 937.584046] Lustre: server umount lustre-MDT0001 complete [ 955.773156] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 958.836760] Key type lgssc unregistered [ 959.215778] LNet: 28344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.220104] LNetError: 28344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.236652] LNet: Removed LNI 192.168.203.121@tcp [ 960.632061] Key type .llcrypt unregistered [ 960.637653] Key type ._llcrypt unregistered [ 995.277297] Key type ._llcrypt registered [ 995.287252] Key type .llcrypt registered [ 995.554599] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 997.010585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.420218] alg: No test for adler32 (adler32-zlib) [ 999.459683] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 999.978736] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 1001.839706] Key type lgssc registered [ 1003.556687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1015.098012] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1015.138775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.878655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.283391] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1027.058228] LustreError: 29760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1027.097465] LustreError: 29760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1032.162209] LustreError: 29761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.210471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.765297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.342638] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1050.462516] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1059.922725] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.705044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1079.153299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.338353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1089.181759] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1102.303376] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1116.131780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.159400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.189163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.153611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.154868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.167679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.234283] Lustre: Skipped 1 previous similar message [ 1122.279974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.293248] Lustre: Skipped 1 previous similar message [ 1126.367404] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1126.506830] Lustre: server umount lustre-OST0000 complete [ 1132.005271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1132.014231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.030780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.519468] Lustre: server umount lustre-MDT0000 complete [ 1137.636627] LustreError: 29761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.659481] LustreError: 29761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1142.767598] LustreError: 29760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.799175] LustreError: 29760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1143.033950] LustreError: 29740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785933302 with bad export cookie 18161187626682208259 [ 1143.051723] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.016189] Lustre: server umount lustre-MDT0001 complete [ 1164.211330] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 1169.709393] Key type lgssc unregistered [ 1170.227413] LNet: 33670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.238506] LNetError: 33670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.278797] LNet: Removed LNI 192.168.203.121@tcp [ 1171.704978] Key type .llcrypt unregistered [ 1171.707918] Key type ._llcrypt unregistered [ 1208.674802] Key type ._llcrypt registered [ 1208.684345] Key type .llcrypt registered [ 1208.830758] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 1210.044692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.089647] alg: No test for adler32 (adler32-zlib) [ 1211.572558] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1212.002348] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 1213.785242] Key type lgssc registered [ 1215.240407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1224.908730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1224.919297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.586588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.206568] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1236.961457] LustreError: 35084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.989172] LustreError: 35084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1242.102632] LustreError: 35085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.208327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.515837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1253.294586] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1263.972388] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.584985] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1282.977298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1283.743554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.066190] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1297.075451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1307.142195] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1317.869613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.893033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.909532] Lustre: Skipped 1 previous similar message [ 1322.977983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.980120] Lustre: Skipped 1 previous similar message [ 1324.668726] Lustre: server umount lustre-OST0000 complete [ 1333.227803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1333.228137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1333.242455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.268636] Lustre: Skipped 1 previous similar message [ 1333.295345] Lustre: Skipped 1 previous similar message [ 1337.054712] Lustre: server umount lustre-MDT0000 complete [ 1338.336951] LustreError: 35080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.379318] LustreError: 35080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1342.993700] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785933502 with bad export cookie 1810467493504595903 [ 1342.994592] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.010549] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1344.291700] Lustre: server umount lustre-MDT0001 complete [ 1365.462062] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 1371.274850] Key type lgssc unregistered [ 1371.611716] LNet: 38649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.624696] LNetError: 38649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.652255] LNet: Removed LNI 192.168.203.121@tcp [ 1372.964834] Key type .llcrypt unregistered [ 1372.971048] Key type ._llcrypt unregistered [ 1403.478455] Key type ._llcrypt registered [ 1403.487652] Key type .llcrypt registered [ 1403.628687] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 1405.551450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.592741] alg: No test for adler32 (adler32-zlib) [ 1406.871565] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1407.321584] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 1409.135130] Key type lgssc registered [ 1410.456373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1419.373748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1419.406955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.912418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.768442] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1430.566406] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.597312] LustreError: 40065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1435.616957] LustreError: 40064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.315106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.971136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.804549] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1452.671801] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1460.379824] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.642822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.290659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.693915] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1485.516195] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1490.814768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1497.058234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.069852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.087482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.174682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1501.176796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.185296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1503.059608] Lustre: server umount lustre-OST0000 complete [ 1526.755089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.786756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.810919] Lustre: Skipped 2 previous similar messages [ 1532.896610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.905375] Lustre: Skipped 4 previous similar messages [ 1533.498969] Lustre: server umount lustre-MDT0000 complete [ 1537.790433] LustreError: 40047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785933697 with bad export cookie 7467156241689581321 [ 1537.798628] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1552.351181] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1552.892508] Lustre: server umount lustre-MDT0001 complete [ 1568.230272] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 1572.936723] Key type lgssc unregistered [ 1573.656508] LNet: 43781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.659989] LNetError: 43781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.714264] LNet: Removed LNI 192.168.203.121@tcp [ 1575.749384] Key type .llcrypt unregistered [ 1575.755669] Key type ._llcrypt unregistered [ 1608.282455] Key type ._llcrypt registered [ 1608.284676] Key type .llcrypt registered [ 1608.443629] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 1611.321647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1611.479543] alg: No test for adler32 (adler32-zlib) [ 1612.746397] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1613.156421] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 1614.943822] Key type lgssc registered [ 1616.852607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1628.445945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.488700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.296508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.431880] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1640.424900] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.460193] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1645.541937] LustreError: 45194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.956583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.396634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1655.616662] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1665.989482] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1673.318687] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1684.157616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1684.908879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.711833] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1700.317499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1710.445523] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1720.802504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.806445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.826950] Lustre: Skipped 1 previous similar message [ 1722.727509] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.21@tcp (stopping) [ 1722.734600] Lustre: Skipped 2 previous similar messages [ 1724.757503] Lustre: server umount lustre-MDT0000 complete [ 1725.934691] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.958788] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1729.824970] LustreError: 45174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785933889 with bad export cookie 11197436232724300482 [ 1729.828886] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1729.839135] LustreError: 45174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1731.040367] LustreError: 45188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.041399] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.066815] LustreError: 45188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1731.106639] Lustre: Skipped 1 previous similar message [ 1731.117525] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1734.048397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1734.055753] Lustre: Skipped 1 previous similar message [ 1738.034908] LustreError: 47202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.056787] LustreError: 47202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1739.233707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1739.241501] Lustre: Skipped 1 previous similar message [ 1744.351111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1744.907376] Lustre: server umount lustre-MDT0001 complete [ 1761.485774] Lustre: server umount lustre-OST0000 complete [ 1786.056420] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 1790.022809] Key type lgssc unregistered [ 1790.511195] LNet: 49065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.531185] LNetError: 49065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.594437] LNet: Removed LNI 192.168.203.121@tcp [ 1793.583787] Key type .llcrypt unregistered [ 1793.586825] Key type ._llcrypt unregistered [ 1826.976447] Key type ._llcrypt registered [ 1826.981740] Key type .llcrypt registered [ 1827.163177] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 1828.803190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.904617] alg: No test for adler32 (adler32-zlib) [ 1830.421160] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 1830.995115] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 1832.703309] Key type lgssc registered [ 1834.675969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.513228] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.555906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.279591] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1866.399483] LustreError: 50466:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 1893.601001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.392920] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 1915.130842] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2084.149063] Lustre: server umount lustre-OST0000 complete [ 2108.441794] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 2111.979245] Key type lgssc unregistered [ 2112.406620] LNet: 52169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.413638] LNetError: 52169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.463295] LNet: Removed LNI 192.168.203.121@tcp [ 2113.340230] Key type .llcrypt unregistered [ 2113.347460] Key type ._llcrypt unregistered [ 2146.781432] Key type ._llcrypt registered [ 2146.785528] Key type .llcrypt registered [ 2146.968211] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 2149.645564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2149.807430] alg: No test for adler32 (adler32-zlib) [ 2150.830793] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 2151.213832] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 2152.895199] Key type lgssc registered [ 2155.459275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2167.042664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2167.083454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.619740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.790933] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2179.041795] LustreError: 53588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.075529] LustreError: 53588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2184.167811] LustreError: 53589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.900095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.421286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2192.372630] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2203.714323] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2214.316995] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2224.143130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.930064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.651376] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2238.285162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2246.969340] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2252.773275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.789953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.819854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.792669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.793259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.800026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2256.609611] Lustre: server umount lustre-OST0000 complete [ 2263.007972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2263.022233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.052848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.055598] Lustre: Skipped 1 previous similar message [ 2268.174539] Lustre: server umount lustre-MDT0000 complete [ 2269.164056] LustreError: 53589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.191864] LustreError: 53589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2273.594882] LustreError: 53568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785934433 with bad export cookie 4832795772019873662 [ 2273.616846] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.005874] Lustre: server umount lustre-MDT0001 complete [ 2288.646435] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 2293.250580] Key type lgssc unregistered [ 2293.734319] LNet: 57124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.760935] LNetError: 57124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.798382] LNet: Removed LNI 192.168.203.121@tcp [ 2295.348099] Key type .llcrypt unregistered [ 2295.352544] Key type ._llcrypt unregistered [ 2327.388438] Key type ._llcrypt registered [ 2327.390919] Key type .llcrypt registered [ 2327.586557] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 2328.729425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.755544] alg: No test for adler32 (adler32-zlib) [ 2330.486992] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 2330.759873] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 2332.519133] Key type lgssc registered [ 2334.803109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.381205] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.444991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2361.247855] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2366.367719] LustreError: 58514:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 2393.057297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.955887] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2411.058896] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.148154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.945355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.037771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2427.372956] LustreError: 58539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.399372] LustreError: 58539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2427.861717] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2428.386688] LustreError: 58540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.416920] LustreError: 58540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2433.508111] LustreError: 58539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.526516] LustreError: 58539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2438.637574] LustreError: 58540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.684980] LustreError: 58540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2439.090324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.921470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2446.904709] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2458.228396] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2466.420463] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2470.368409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2470.381805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.408495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2475.909957] Lustre: server umount lustre-OST0000 complete [ 2505.698079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.728943] Lustre: Skipped 1 previous similar message [ 2505.743436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2505.758554] Lustre: Skipped 1 previous similar message [ 2510.126602] Lustre: server umount lustre-MDT0000 complete [ 2510.819674] LustreError: 60058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.845284] LustreError: 60058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2514.980548] LustreError: 58522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785934674 with bad export cookie 7928720037314990581 [ 2514.982449] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.004960] LustreError: 58522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2521.698854] Lustre: server umount lustre-MDT0001 complete [ 2541.938537] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 2546.270248] Key type lgssc unregistered [ 2546.738143] LNet: 62290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.742506] LNetError: 62290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.775425] LNet: Removed LNI 192.168.203.121@tcp [ 2548.424444] Key type .llcrypt unregistered [ 2548.429839] Key type ._llcrypt unregistered [ 2580.327246] Key type ._llcrypt registered [ 2580.333995] Key type .llcrypt registered [ 2580.474212] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 2581.978270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2582.036912] alg: No test for adler32 (adler32-zlib) [ 2583.308813] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 2583.537912] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 2585.231248] Key type lgssc registered [ 2586.329201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2597.006025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2597.032409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.511451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.316874] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2609.138162] LustreError: 63703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.183636] LustreError: 63703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2614.243495] LustreError: 63704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.893606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.290738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2624.998758] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2634.432802] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2641.026278] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2648.358077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2648.541772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.717891] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2669.118978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2675.413979] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2699.745774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.748784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.751740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.786458] Lustre: Skipped 1 previous similar message [ 2699.817879] Lustre: Skipped 1 previous similar message [ 2704.873342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.889618] Lustre: Skipped 1 previous similar message [ 2706.018426] Lustre: server umount lustre-OST0000 complete [ 2711.520853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2711.528547] LustreError: Skipped 1 previous similar message [ 2711.534185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.552036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.105269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.107209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.686895] Lustre: server umount lustre-MDT0000 complete [ 2720.227878] LustreError: 63704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.253060] LustreError: 63704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2722.149608] LustreError: 63682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785934881 with bad export cookie 15385925941682876613 [ 2722.176218] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.008381] Lustre: server umount lustre-MDT0001 complete [ 2737.182414] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 2741.247503] Key type lgssc unregistered [ 2741.799905] LNet: 67277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2741.812556] LNetError: 67277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2741.844868] LNet: Removed LNI 192.168.203.121@tcp [ 2743.203304] Key type .llcrypt unregistered [ 2743.206861] Key type ._llcrypt unregistered [ 2779.499620] Key type ._llcrypt registered [ 2779.502542] Key type .llcrypt registered [ 2779.667435] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 2781.360591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2781.530619] alg: No test for adler32 (adler32-zlib) [ 2782.855143] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 2783.166337] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 2784.991391] Key type lgssc registered [ 2787.145802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2801.016856] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:02:38 (1785934958) [ 2810.307577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2810.340515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.142822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.313721] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2821.602099] LustreError: 68985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.623386] LustreError: 68985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2826.733932] LustreError: 68986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.591035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2830.241167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.287754] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2846.466903] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2856.121489] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2864.015465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2864.298382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.066702] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2881.711768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2882.022228] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2890.361502] Lustre: Failing over lustre-MDT0000 [ 2890.962908] Lustre: server umount lustre-MDT0000 complete [ 2891.756459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2891.764930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.804878] LustreError: 68986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.259058] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.295334] Lustre: Skipped 1 previous similar message [ 2895.208878] LustreError: 69722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.231927] LustreError: 69722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2900.284177] LustreError: 68982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.318463] LustreError: 68982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2905.653963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.015432] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.467747] Lustre: *** cfs_fail_loc=135, val=0*** [ 2906.888708] LustreError: 72076:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2906.909770] LustreError: 72076:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.121@tcp: cfg command failed: rc = -2 [ 2906.933917] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2906.933917] [ 2906.943900] LustreError: MGC192.168.203.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2906.992936] LustreError: 72063:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2907.008706] LustreError: 72063:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2907.041640] LustreError: 72063:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2907.071741] LustreError: 72063:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2907.142858] Lustre: server umount lustre-MDT0000 complete [ 2907.153153] LustreError: 72063:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2910.512183] LustreError: 68982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.534932] LustreError: 68982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2919.372848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.665545] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.679355] LustreError: 68978:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2920.156394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.245431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2921.779581] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2925.554247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2925.588151] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2925.623261] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 2925.675606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2940.928156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.937126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.976443] Lustre: Skipped 1 previous similar message [ 2944.421201] Lustre: server umount lustre-OST0000 complete [ 2951.136029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2951.137779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.154335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.179075] Lustre: Skipped 1 previous similar message [ 2951.201958] Lustre: Skipped 2 previous similar messages [ 2955.140233] Lustre: server umount lustre-MDT0000 complete [ 2956.258166] LustreError: 68986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.292963] LustreError: 68986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 2960.417429] LustreError: 68965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935120 with bad export cookie 15382085280757304835 [ 2960.449025] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.958265] Lustre: server umount lustre-MDT0001 complete [ 2976.158482] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 2980.007444] Key type lgssc unregistered [ 2980.412269] LNet: 73861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.436464] LNetError: 73861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2980.468052] LNet: Removed LNI 192.168.203.121@tcp [ 2981.858159] Key type .llcrypt unregistered [ 2981.861935] Key type ._llcrypt unregistered [ 3015.182406] Key type ._llcrypt registered [ 3015.197501] Key type .llcrypt registered [ 3015.375378] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 3017.064827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3017.092158] alg: No test for adler32 (adler32-zlib) [ 3018.115990] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3018.456488] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 3020.151119] Key type lgssc registered [ 3022.656933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.819820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.846730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.803525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.992717] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3047.461720] LustreError: 75275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.514790] LustreError: 75275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3052.516522] LustreError: 75276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.658594] LustreError: 75275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.264856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.958642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.603917] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3078.726141] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3087.345584] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3097.150813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.757291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3106.159574] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3108.116989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3117.848714] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3126.165210] Lustre: Failing over lustre-MDT0000 [ 3126.635927] Lustre: server umount lustre-MDT0000 complete [ 3127.264938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3127.286386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.318979] LustreError: 75276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.805665] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.851801] Lustre: Skipped 1 previous similar message [ 3133.923665] LustreError: 76004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.946095] LustreError: 76004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3141.780233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.134783] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.651786] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3142.940979] LustreError: 78365:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3142.961210] LustreError: 78365:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.121@tcp: cfg command failed: rc = -9 [ 3142.984510] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3142.984510] [ 3143.013382] LustreError: MGC192.168.203.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3143.061704] LustreError: 78354:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3143.093297] LustreError: 78354:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3143.108532] LustreError: 78354:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3143.129745] LustreError: 78354:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3143.202746] Lustre: server umount lustre-MDT0000 complete [ 3143.211794] LustreError: 78354:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3144.169863] LustreError: 75271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.194218] LustreError: 75271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3154.724423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.126714] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.149079] LustreError: 75268:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3155.704603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.804820] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3156.786703] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3161.087756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3161.130930] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3161.187908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3162.324595] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3176.439621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.473364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.495562] Lustre: Skipped 1 previous similar message [ 3181.545211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.664610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3186.677700] Lustre: Skipped 2 previous similar messages [ 3189.215145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3189.709946] Lustre: server umount lustre-OST0000 complete [ 3196.895946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.898468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3196.905062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.922158] Lustre: Skipped 2 previous similar messages [ 3202.021384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3202.024615] Lustre: Skipped 1 previous similar message [ 3209.184307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3209.987293] Lustre: server umount lustre-MDT0000 complete [ 3212.258901] LustreError: 76004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.279442] LustreError: 76004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3215.521668] LustreError: 75256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935375 with bad export cookie 4944358812093946100 [ 3215.533422] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.543458] LustreError: 75256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3216.013235] Lustre: server umount lustre-MDT0001 complete [ 3232.632163] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 3238.487509] Key type lgssc unregistered [ 3239.225172] LNet: 80155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3239.251560] LNetError: 80155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3239.296402] LNet: Removed LNI 192.168.203.121@tcp [ 3241.250718] Key type .llcrypt unregistered [ 3241.261834] Key type ._llcrypt unregistered [ 3261.946590] Key type ._llcrypt registered [ 3261.949489] Key type .llcrypt registered [ 3262.044625] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 3262.371348] Key type .llcrypt unregistered [ 3262.373390] Key type ._llcrypt unregistered [ 3300.022938] Key type ._llcrypt registered [ 3300.029734] Key type .llcrypt registered [ 3300.172680] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 3301.761726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3301.804792] alg: No test for adler32 (adler32-zlib) [ 3303.210725] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3303.694969] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 3305.471276] Key type lgssc registered [ 3307.790860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3318.441453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3318.473101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.206465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.703603] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3329.510168] LustreError: 82068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.560363] LustreError: 82068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3334.625416] LustreError: 82069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.998794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.715497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.888545] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3357.248962] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3364.576753] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3374.601120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3375.159917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.037091] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3389.650397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3396.301975] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3415.010801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.018871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.048349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.520361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.562910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3418.293567] Lustre: server umount lustre-OST0000 complete [ 3425.760519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.766196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.790889] Lustre: Skipped 1 previous similar message [ 3425.808698] Lustre: Skipped 1 previous similar message [ 3430.505222] Lustre: server umount lustre-MDT0000 complete [ 3430.904223] LustreError: 82797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3430.943020] LustreError: 82797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3436.001571] LustreError: 82069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.032650] LustreError: 82069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3436.745079] LustreError: 82050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935596 with bad export cookie 5640568705850637138 [ 3436.770783] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.794753] LustreError: 82050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3437.279141] Lustre: server umount lustre-MDT0001 complete [ 3453.348242] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 3458.264767] Key type lgssc unregistered [ 3458.859837] LNet: 85633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3458.872102] LNetError: 85633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3458.908385] LNet: Removed LNI 192.168.203.121@tcp [ 3460.981906] Key type .llcrypt unregistered [ 3460.983760] Key type ._llcrypt unregistered [ 3496.685311] Key type ._llcrypt registered [ 3496.687463] Key type .llcrypt registered [ 3496.777854] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 3498.531384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.595054] alg: No test for adler32 (adler32-zlib) [ 3499.646574] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3499.978272] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 3501.775461] Key type lgssc registered [ 3503.652633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3514.641877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3514.666973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.331417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.579465] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3526.629293] LustreError: 87047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.657590] LustreError: 87047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3531.744277] LustreError: 87048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.542701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.143074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3540.341333] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3551.579761] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3559.393073] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3567.392022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.869807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.692104] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3586.318137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3586.930734] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3596.784016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.787076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3596.828814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.583562] Lustre: server umount lustre-OST0000 complete [ 3604.960704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3604.970430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.989973] Lustre: Skipped 1 previous similar message [ 3604.993925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.996537] Lustre: Skipped 1 previous similar message [ 3607.012870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.014154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.048200] Lustre: Skipped 1 previous similar message [ 3610.838334] Lustre: server umount lustre-MDT0000 complete [ 3612.143678] LustreError: 87048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.174786] LustreError: 87048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3616.290233] LustreError: 87047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3616.575798] LustreError: 87028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935776 with bad export cookie 6414409795457295550 [ 3616.586397] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.592335] LustreError: 87028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3617.457829] Lustre: server umount lustre-MDT0001 complete [ 3633.103654] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 3636.425898] Key type lgssc unregistered [ 3636.847784] LNet: 90585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3636.862627] LNetError: 90585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3637.930072] LNet: Removed LNI 192.168.203.121@tcp [ 3639.010136] Key type .llcrypt unregistered [ 3639.014166] Key type ._llcrypt unregistered [ 3671.195050] Key type ._llcrypt registered [ 3671.198413] Key type .llcrypt registered [ 3671.362802] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 3673.306071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3673.360152] alg: No test for adler32 (adler32-zlib) [ 3674.657483] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3675.144611] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 3676.927167] Key type lgssc registered [ 3678.593469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3688.965110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3689.006155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.677427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.821803] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3701.224025] LustreError: 92000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.255555] LustreError: 92000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3706.345728] LustreError: 92001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.671144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.159143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3716.187745] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3725.688566] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3734.181544] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3744.350146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3744.818437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.853090] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3762.234481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3765.986934] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3787.763598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3787.769644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.803109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.849795] Lustre: Skipped 1 previous similar message [ 3787.880927] Lustre: Skipped 1 previous similar message [ 3792.476371] Lustre: server umount lustre-OST0000 complete [ 3800.546194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3800.566176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.587624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.109457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.111551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3805.116444] Lustre: server umount lustre-MDT0000 complete [ 3807.266873] LustreError: 91997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.307283] LustreError: 91997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3810.867979] LustreError: 94506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785935970 with bad export cookie 5657964790563392008 [ 3810.877756] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.892203] LustreError: 94506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3811.271063] Lustre: server umount lustre-MDT0001 complete [ 3831.563680] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 3837.083195] Key type lgssc unregistered [ 3837.244484] LNet: 95584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3837.247965] LNetError: 95584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3838.322828] LNet: Removed LNI 192.168.203.121@tcp [ 3840.118101] Key type .llcrypt unregistered [ 3840.121347] Key type ._llcrypt unregistered [ 3870.956820] Key type ._llcrypt registered [ 3870.960117] Key type .llcrypt registered [ 3871.146689] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 3872.710952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3872.751614] alg: No test for adler32 (adler32-zlib) [ 3873.761546] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 3874.069893] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 3875.815159] Key type lgssc registered [ 3877.680415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3889.688705] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3889.720473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3905.439542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3910.560077] LustreError: 96977:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 3937.255862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.202852] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3957.850796] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3967.478337] Lustre: server umount lustre-OST0000 complete [ 3978.579246] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:22:15 (1785936135) [ 3988.115759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.819521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.079895] LustreError: 98617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.114916] LustreError: 98617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3995.620882] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 3999.210393] LustreError: 98616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.323227] LustreError: 98617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.460366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.962492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4014.099813] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4026.714038] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4034.811754] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4044.186246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4044.824930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.637328] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4059.215086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4066.581846] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4082.657428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.677390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.726031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.704220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.710426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.727720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.756220] Lustre: Skipped 1 previous similar message [ 4088.131696] Lustre: server umount lustre-OST0000 complete [ 4099.042417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4099.064669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.120417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.427931] Lustre: server umount lustre-MDT0000 complete [ 4105.200036] LustreError: 98611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.258961] LustreError: 98611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4108.398918] LustreError: 99352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936268 with bad export cookie 12108123929481850596 [ 4108.412604] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4108.414948] LustreError: 99352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4109.197561] Lustre: server umount lustre-MDT0001 complete [ 4124.336258] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 4128.248646] Key type lgssc unregistered [ 4128.983295] LNet: 102316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4129.010991] LNetError: 102316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4129.051259] LNet: Removed LNI 192.168.203.121@tcp [ 4130.806309] Key type .llcrypt unregistered [ 4130.812766] Key type ._llcrypt unregistered [ 4195.864771] Key type ._llcrypt registered [ 4195.867203] Key type .llcrypt registered [ 4195.967916] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 4197.687731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4197.793193] alg: No test for adler32 (adler32-zlib) [ 4198.932409] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 4199.130537] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 4200.791671] Key type lgssc registered [ 4202.675639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4213.272566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4213.288163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.800657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4220.158548] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4225.008054] LustreError: 104504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.033405] LustreError: 104504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4230.115194] LustreError: 104505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.754577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.406563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4238.149647] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4249.845029] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4258.955811] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4269.830649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.055099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.599370] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4284.406613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4287.346793] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4298.722371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4298.751049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.789542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.744029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.748509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4300.405936] Lustre: server umount lustre-OST0000 complete [ 4309.480481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4309.487401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.516110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4309.534805] Lustre: Skipped 1 previous similar message [ 4314.882743] Lustre: server umount lustre-MDT0000 complete [ 4315.114126] LustreError: 105455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.165960] LustreError: 105455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4320.236326] LustreError: 104504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.273527] LustreError: 104504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4322.537719] LustreError: 104486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936482 with bad export cookie 911911113453599155 [ 4322.560402] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.564491] LustreError: 104486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4323.644373] Lustre: server umount lustre-MDT0001 complete [ 4341.066464] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 4345.539185] Key type lgssc unregistered [ 4346.012400] LNet: 108043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4346.024664] LNetError: 108043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4346.046193] LNet: Removed LNI 192.168.203.121@tcp [ 4347.114154] Key type .llcrypt unregistered [ 4347.118122] Key type ._llcrypt unregistered [ 4374.017033] Key type ._llcrypt registered [ 4374.018503] Key type .llcrypt registered [ 4374.135940] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 4375.918161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4376.187757] alg: No test for adler32 (adler32-zlib) [ 4377.440152] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 4377.718889] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 4379.624425] Key type lgssc registered [ 4381.661243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4392.565341] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4392.604883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4408.287364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4413.409219] LustreError: 109288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 4440.159212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.090451] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4460.675322] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4470.131924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.500863] LustreError: MGC192.168.203.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4470.513446] LustreError: 110314:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4470.523405] LustreError: 110314:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4470.539276] LustreError: 110314:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4470.555536] LustreError: 109296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936630 with bad export cookie 9494679166639699026 [ 4470.581279] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4470.685737] Lustre: server umount lustre-MDT0000 complete [ 4470.690396] LustreError: 110314:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4498.115458] Lustre: server umount lustre-OST0000 complete [ 4514.919781] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 4528.242654] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 4582.771009] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.259940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4583.355190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4583.558448] Lustre: lustre-MDT0000: new disk, initializing [ 4583.769227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.812811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4591.061395] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4604.257305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.428567] Lustre: 113110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4604.509792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4604.517782] Lustre: Skipped 1 previous similar message [ 4604.643813] Lustre: lustre-MDT0001: new disk, initializing [ 4604.769721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4604.845228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4604.861041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4611.836880] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4624.799361] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4634.284611] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4642.994441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.476721] Lustre: lustre-OST0000: new disk, initializing [ 4643.482829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4643.495761] Lustre: 115080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4643.650784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.581345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4650.600504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4650.738074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4651.953654] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4665.334829] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4671.683936] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4671.925345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4678.629912] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4678.909870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4681.708480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4681.712719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.723168] LustreError: Skipped 1 previous similar message [ 4681.728187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4681.752330] Lustre: Skipped 1 previous similar message [ 4686.821343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4686.841978] Lustre: Skipped 2 previous similar messages [ 4688.104940] Lustre: server umount lustre-OST0000 complete [ 4697.056255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.057987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.085069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.115463] Lustre: Skipped 1 previous similar message [ 4699.606638] Lustre: server umount lustre-MDT0000 complete [ 4704.685693] LustreError: 113104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785936864 with bad export cookie 9494679166639699131 [ 4704.697650] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4704.698816] LustreError: 113104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4705.532105] Lustre: server umount lustre-MDT0001 complete [ 4718.153163] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:34:34 (1785936874) [ 4723.473154] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4747.667806] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 4758.348067] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 4807.013182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.322513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4807.368447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4807.486392] Lustre: lustre-MDT0000: new disk, initializing [ 4807.614622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.630348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4813.501705] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4827.775205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.921411] Lustre: 119991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4828.003122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4828.013370] Lustre: Skipped 1 previous similar message [ 4828.162521] Lustre: lustre-MDT0001: new disk, initializing [ 4828.344144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4828.412171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4828.431753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4834.868344] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4850.099630] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.777762] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4870.448746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4870.661926] Lustre: lustre-OST0000: new disk, initializing [ 4870.664670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4870.669849] Lustre: 121962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4870.734506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4879.979412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4879.998036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4880.209806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4880.506810] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4893.012311] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4899.344463] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4899.657764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4905.008681] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4905.359681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4911.076673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.095667] Lustre: Skipped 1 previous similar message [ 4911.100761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4911.105646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4916.195888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4916.209607] Lustre: Skipped 1 previous similar message [ 4922.336344] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4922.646553] Lustre: server umount lustre-OST0000 complete [ 4930.534059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4930.543614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.563730] Lustre: Skipped 1 previous similar message [ 4930.576294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.579051] Lustre: Skipped 3 previous similar messages [ 4933.950563] Lustre: server umount lustre-MDT0000 complete [ 4936.694139] LustreError: 121307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.713815] LustreError: 121307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4938.907062] LustreError: 119984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937098 with bad export cookie 9494679166639700013 [ 4938.929149] LustreError: 119984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4938.932439] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4939.872931] Lustre: server umount lustre-MDT0001 complete [ 4952.464215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.892795] LustreError: 124181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.968198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4958.182511] LustreError: 124182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.105707] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4962.339831] LustreError: 124181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.406820] LustreError: 124182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.731053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.250941] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 4990.591450] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4999.288513] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5007.837600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5016.908698] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5027.152450] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5037.570375] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5043.170991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.182050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.195483] Lustre: Skipped 1 previous similar message [ 5043.210194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.226353] Lustre: Skipped 2 previous similar messages [ 5049.744078] Lustre: server umount lustre-OST0000 complete [ 5058.530013] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.554634] Lustre: Skipped 2 previous similar messages [ 5071.327152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5071.763057] Lustre: server umount lustre-MDT0000 complete [ 5073.890633] LustreError: 124177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.919819] LustreError: 124177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5078.059326] LustreError: 124162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937237 with bad export cookie 9494679166639700895 [ 5078.081468] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.085264] LustreError: 124162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5078.655284] Lustre: server umount lustre-MDT0001 complete [ 5098.159249] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 5103.209531] Key type lgssc unregistered [ 5103.817511] LNet: 127840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5103.827992] LNetError: 127840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5103.875823] LNet: Removed LNI 192.168.203.121@tcp [ 5105.464910] Key type .llcrypt unregistered [ 5105.467861] Key type ._llcrypt unregistered [ 5130.589525] Key type ._llcrypt registered [ 5130.597923] Key type .llcrypt registered [ 5130.785290] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 5151.610992] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 5153.593886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5153.661832] alg: No test for adler32 (adler32-zlib) [ 5154.827171] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 5155.150592] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 5156.815650] Key type lgssc registered [ 5158.417555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5208.630361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5208.675869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.104199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5210.181402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5210.367532] Lustre: lustre-MDT0000: new disk, initializing [ 5210.526288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.572944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5217.069252] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5230.608361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.766681] Lustre: 131002:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5230.825840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5230.830629] Lustre: Skipped 1 previous similar message [ 5230.902918] Lustre: lustre-MDT0001: new disk, initializing [ 5230.991380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5231.039598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5231.062690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5237.686899] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5251.036037] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5258.375889] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5269.517119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5270.277805] Lustre: lustre-OST0000: new disk, initializing [ 5270.287648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5270.304202] Lustre: 132973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5270.467064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.371973] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5280.472165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5280.499927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5280.667377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5292.539175] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5298.315823] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5298.537349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5304.408435] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5304.751834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5311.456835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.463313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.510139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5311.553616] Lustre: Skipped 1 previous similar message [ 5313.659205] Lustre: server umount lustre-OST0000 complete [ 5321.701257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.706314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.747953] Lustre: Skipped 1 previous similar message [ 5321.750578] Lustre: Skipped 1 previous similar message [ 5325.820809] Lustre: server umount lustre-MDT0000 complete [ 5331.965496] LustreError: 131011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.015922] LustreError: 131011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5332.775570] LustreError: 130996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937492 with bad export cookie 5808204149115653480 [ 5332.793339] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.804015] LustreError: 130996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5334.073853] Lustre: server umount lustre-MDT0001 complete [ 5349.983347] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:45:06 (1785937506) [ 5361.366664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.906155] LustreError: 135387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5362.013656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5367.264525] LustreError: 135388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.432784] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5371.380127] LustreError: 135387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5376.485828] LustreError: 135388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.187930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.860957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5389.480245] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5403.372477] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5412.645965] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5417.953421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.970075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5417.970712] Lustre: Skipped 1 previous similar message [ 5417.989184] Lustre: Skipped 2 previous similar messages [ 5422.170866] Lustre: server umount lustre-MDT0000 complete [ 5423.076106] LustreError: 136289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.099501] LustreError: 136289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5426.132452] LustreError: 135368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937585 with bad export cookie 5808204149115654362 [ 5426.143450] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.150558] LustreError: 135368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5426.791891] Lustre: server umount lustre-MDT0001 complete [ 5437.223167] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:46:34 (1785937594) [ 5446.003349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5461.727562] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5466.847892] LustreError: 137807:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 5492.577261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5501.716634] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5512.711144] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5516.320837] Lustre: server umount lustre-OST0000 complete [ 5528.567119] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:48:06 (1785937686) [ 5537.378216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.130293] LustreError: 139257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5538.300112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5544.792531] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5556.620334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.130865] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5571.969248] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5580.334119] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5589.579070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5590.095302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5590.107507] Lustre: Skipped 1 previous similar message [ 5598.487757] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5608.500572] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5618.691919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5618.714263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.743934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.289324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.305350] Lustre: Skipped 5 previous similar messages [ 5625.384718] Lustre: server umount lustre-MDT0000 complete [ 5629.415130] LustreError: 139257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.452739] LustreError: 139257:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5630.197533] LustreError: 139237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937789 with bad export cookie 5808204149115655027 [ 5630.214957] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.216895] LustreError: 139237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5630.686441] Lustre: server umount lustre-MDT0001 complete [ 5646.057489] Lustre: server umount lustre-OST0000 complete [ 5656.966286] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:50:14 (1785937814) [ 5665.085437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.646048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.124807] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5681.500181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.735282] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5699.320950] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5710.533970] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5721.038785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5729.505437] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5735.932363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5740.385354] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5747.965535] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5748.177276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5755.303297] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5755.504903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5761.516664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5761.519651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.524042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5761.551308] Lustre: Skipped 3 previous similar messages [ 5772.256155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5772.848683] Lustre: server umount lustre-OST0000 complete [ 5779.467221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5779.483208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.507972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.523325] Lustre: Skipped 5 previous similar messages [ 5785.643445] Lustre: server umount lustre-MDT0000 complete [ 5787.110377] LustreError: 142922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.124098] LustreError: 142922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5790.177934] LustreError: 142902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785937949 with bad export cookie 5808204149115656098 [ 5790.180153] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5790.193884] LustreError: 142902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5790.723046] Lustre: server umount lustre-MDT0001 complete [ 5805.222178] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:52:42 (1785937962) [ 5816.500140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5832.288553] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5837.407424] LustreError: 147090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 5863.137859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5863.146810] Lustre: Skipped 2 previous similar messages [ 5870.673207] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5881.167047] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5890.186422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5890.911977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5897.744631] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5911.003793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.006563] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 5932.106624] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5944.733749] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5949.997117] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5950.263218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5956.510817] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5956.857030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5962.720106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5962.725623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.740302] Lustre: Skipped 1 previous similar message [ 5962.750671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5962.761845] Lustre: Skipped 2 previous similar messages [ 5965.931749] Lustre: server umount lustre-MDT0000 complete [ 5968.365817] LustreError: 149077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5968.382223] LustreError: 149077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5971.207108] LustreError: 147097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938130 with bad export cookie 5808204149115657120 [ 5971.211754] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5971.227316] LustreError: 147097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5971.902382] Lustre: server umount lustre-MDT0001 complete [ 5989.311961] Lustre: server umount lustre-OST0000 complete [ 6006.300715] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:56:02 (1785938162) [ 6015.770696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6031.455701] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6036.575486] LustreError: 151291:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 6070.640235] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6084.277165] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6093.315647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.333980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6100.157512] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6111.435188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.546398] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6131.614692] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6141.238375] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6151.066148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6151.415194] Lustre: lustre-OST0001: new disk, initializing [ 6151.421770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6151.439828] Lustre: 154167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6151.544883] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6151.553851] Lustre: Skipped 5 previous similar messages [ 6157.439289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6157.462657] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6157.691076] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6160.427570] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6173.334230] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6180.331543] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6180.640485] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6187.897949] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6188.456880] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6191.584623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6191.603110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.629834] Lustre: Skipped 2 previous similar messages [ 6191.645063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6191.647786] Lustre: Skipped 3 previous similar messages [ 6197.313405] Lustre: server umount lustre-OST0000 complete [ 6209.166678] Lustre: server umount lustre-OST0001 complete [ 6214.119555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6214.129284] LustreError: Skipped 3 previous similar messages [ 6220.654781] Lustre: server umount lustre-MDT0000 complete [ 6226.055675] LustreError: 151299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938385 with bad export cookie 5808204149115658072 [ 6226.068526] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6226.075939] LustreError: 151299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6226.635721] Lustre: server umount lustre-MDT0001 complete [ 6264.969750] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:00:21 (1785938421) [ 6267.783136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6270.864575] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6273.274458] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:00:30 (1785938430) [ 6282.822732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.534092] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6283.656993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6290.729707] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6303.269203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.505065] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6303.566933] Lustre: 157737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6310.202478] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6319.440584] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6328.217765] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6336.719468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.017961] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6344.794299] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6346.959095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6353.805712] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.394083] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6360.742845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6366.289541] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6366.650042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6372.320641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6372.332927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.378575] Lustre: Skipped 5 previous similar messages [ 6372.407558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.413959] Lustre: Skipped 8 previous similar messages [ 6375.390164] Lustre: server umount lustre-OST0000 complete [ 6402.068897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6411.993409] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6413.108340] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.136225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.141501] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6423.760933] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.893107] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6432.263330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6439.748477] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6440.175731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6449.583645] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff947791c90000.ost_server_uuid 50 [ 6451.690326] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff947791c90000.ost_server_uuid in IDLE state after 0 sec [ 6451.938217] LustreError: 157743:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.21@tcp) returned error from blocking AST (req@000000000767f54e x1872691005272320 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000054f93073/0x509ae2e682fd28b2 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.21@tcp remote: 0xcd1de34532dce7a4 expref: 5 pid: 157743 timeout: 6551 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6451.963460] LustreError: lustre-MDT0000: A client on nid 192.168.203.21@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6451.969666] LustreError: 157733:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.21@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000054f93073/0x509ae2e682fd28b2 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.21@tcp remote: 0xcd1de34532dce7a4 expref: 6 pid: 157743 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6459.362113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.373237] LustreError: Skipped 1 previous similar message [ 6465.674679] Lustre: server umount lustre-OST0000 complete [ 6477.018934] Lustre: server umount lustre-MDT0000 complete [ 6479.840599] LustreError: 159301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.867116] LustreError: 159301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 6482.081026] LustreError: 157730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938641 with bad export cookie 5808204149115659255 [ 6482.090248] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.095821] LustreError: 157730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6488.866783] Lustre: server umount lustre-MDT0001 complete [ 6503.413645] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 6506.805093] Key type lgssc unregistered [ 6507.202719] LNet: 163871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6507.217088] LNetError: 163871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6507.237206] LNet: Removed LNI 192.168.203.121@tcp [ 6508.307175] Key type .llcrypt unregistered [ 6508.311657] Key type ._llcrypt unregistered [ 6538.050209] Key type ._llcrypt registered [ 6538.053479] Key type .llcrypt registered [ 6538.198544] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 6539.794922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6539.852174] alg: No test for adler32 (adler32-zlib) [ 6541.160981] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 6541.419149] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 6543.087163] Key type lgssc registered [ 6544.286595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6552.570080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6552.591298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.237300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6559.664927] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6563.810036] LustreError: 165285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.840731] LustreError: 165285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6568.933356] LustreError: 165286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.070333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.544351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6577.120713] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6587.712936] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6597.339631] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6605.813567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6606.428938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.705579] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6623.889534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6626.878248] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6633.660466] Lustre: Failing over lustre-MDT0000 [ 6634.406830] Lustre: server umount lustre-MDT0000 complete [ 6634.474304] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.479975] LustreError: 165281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.507955] Lustre: Skipped 1 previous similar message [ 6634.567755] LustreError: 165281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6639.596804] LustreError: 165285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.620031] LustreError: 165285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6644.705798] LustreError: 166206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.726922] LustreError: 166206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6647.237719] LustreError: 165266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.21@tcp arrived at 1785938807 with bad export cookie 11703669089927651567 [ 6647.253102] LustreError: 165266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6650.867931] Lustre: 164599:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785938794/real 1785938794] req@0000000027021518 x1872692459369728/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785938810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6650.898139] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.985369] LustreError: 165285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6656.017368] LustreError: 165285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6656.185364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.602546] LustreError: 168341:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6661.609203] LustreError: 168341:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000e5811152 x1872692459373568/t0(0) o250->MGC192.168.203.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785938820 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6661.641799] LustreError: 168341:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6662.134958] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa26bc90751db6151 [ 6662.154124] Lustre: MGC192.168.203.121@tcp: Connection restored to 0@lo (at 0@lo) [ 6662.679576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6662.768482] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6667.750925] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6667.758289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6668.689095] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6671.475169] Lustre: lustre-MDT0000: Denying connection for new client 3ba22620-7bf3-4dda-bf1f-03b49c30affa (at 192.168.203.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 6682.592885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6682.596204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.618336] Lustre: Skipped 1 previous similar message [ 6682.640014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6688.229014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6688.232927] Lustre: Skipped 2 previous similar messages [ 6693.344144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6693.349524] Lustre: Skipped 1 previous similar message [ 6694.367247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6694.768786] Lustre: server umount lustre-OST0000 complete [ 6699.041576] LustreError: 169026:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6699.055456] Lustre: 168373:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6699.065903] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 6699.082478] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 6699.095925] LustreError: 168373:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005d30a9b5 x1872692459393024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6699.112199] LustreError: 168373:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6699.133084] LustreError: 168373:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 6699.156469] Lustre: lustre-MDT0000: Recovery over after 0:32, of 2 clients 0 recovered and 2 were evicted. [ 6699.531150] Lustre: server umount lustre-MDT0000 complete [ 6703.584324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.611600] Lustre: Skipped 1 previous similar message [ 6703.630793] LustreError: 165280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6703.671982] LustreError: 165280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6704.329871] LustreError: 165265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785938864 with bad export cookie 11703669089927651665 [ 6704.346736] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6704.347622] LustreError: 165265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6705.249756] Lustre: server umount lustre-MDT0001 complete [ 6720.730922] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 6724.385723] Key type lgssc unregistered [ 6724.848498] LNet: 169781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6724.880618] LNetError: 169781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6725.931501] LNet: Removed LNI 192.168.203.121@tcp [ 6727.377802] Key type .llcrypt unregistered [ 6727.381883] Key type ._llcrypt unregistered [ 6761.935310] Key type ._llcrypt registered [ 6761.937585] Key type .llcrypt registered [ 6762.019688] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 6763.049735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6763.086941] alg: No test for adler32 (adler32-zlib) [ 6764.600895] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 6765.075579] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 6766.943201] Key type lgssc registered [ 6768.917687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6780.868195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6780.882586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.438772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6789.368781] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6792.673250] LustreError: 171211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.709557] LustreError: 171211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6797.803758] LustreError: 171212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.745521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.793143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6807.508856] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6817.735613] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6825.597947] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6833.927048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6834.407391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.525524] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 6852.844072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6853.767248] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6862.305451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6862.319046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.339805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.329445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.333883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.340041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.287554] Lustre: server umount lustre-OST0000 complete [ 6873.576715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6873.577087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.583280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6873.583287] Lustre: Skipped 1 previous similar message [ 6873.626316] Lustre: Skipped 1 previous similar message [ 6878.692377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.702921] Lustre: Skipped 2 previous similar messages [ 6883.813343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6883.833413] Lustre: Skipped 1 previous similar message [ 6887.393705] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6887.753162] Lustre: server umount lustre-MDT0000 complete [ 6888.936995] LustreError: 171940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.989902] LustreError: 171940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6894.048100] LustreError: 171211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.098247] LustreError: 171211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6895.606436] LustreError: 171947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939055 with bad export cookie 12126092346714736326 [ 6895.633344] LustreError: 171947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6895.636259] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6897.144606] Lustre: server umount lustre-MDT0001 complete [ 6925.114802] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 6930.686033] Key type lgssc unregistered [ 6931.236683] LNet: 174775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6931.247357] LNetError: 174775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6931.265463] LNet: Removed LNI 192.168.203.121@tcp [ 6932.633217] Key type .llcrypt unregistered [ 6932.636828] Key type ._llcrypt unregistered [ 6974.314527] Key type ._llcrypt registered [ 6974.317641] Key type .llcrypt registered [ 6974.540318] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 6976.505903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6976.554737] alg: No test for adler32 (adler32-zlib) [ 6977.718949] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 6978.018561] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 6979.703145] Key type lgssc registered [ 6981.289629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6990.701513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6990.729938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.243755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6997.570253] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7001.574343] LustreError: 176480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.597976] LustreError: 176480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7006.694212] LustreError: 176479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7008.948199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.621097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7015.965899] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7025.764802] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7034.223376] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7044.024136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7044.528201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7051.213499] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7061.803107] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.969406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7077.369701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7077.376543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.386042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7077.403764] Lustre: Skipped 1 previous similar message [ 7077.428125] Lustre: Skipped 1 previous similar message [ 7082.486772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7082.505454] Lustre: Skipped 1 previous similar message [ 7083.046145] Lustre: server umount lustre-OST0000 complete [ 7091.685269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7091.702891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.734529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7092.713185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.342874] Lustre: server umount lustre-MDT0000 complete [ 7097.826506] LustreError: 177207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.851349] LustreError: 177207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7102.949762] LustreError: 176480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.996329] LustreError: 176480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7103.186784] LustreError: 176459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939262 with bad export cookie 2410917929341345187 [ 7103.213773] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.224083] LustreError: 176459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7104.483393] Lustre: server umount lustre-MDT0001 complete [ 7121.294884] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 7125.374674] Key type lgssc unregistered [ 7125.893638] LNet: 180041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7125.902065] LNetError: 180041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7125.940570] LNet: Removed LNI 192.168.203.121@tcp [ 7127.235545] Key type .llcrypt unregistered [ 7127.239412] Key type ._llcrypt unregistered [ 7174.032642] Key type ._llcrypt registered [ 7174.041641] Key type .llcrypt registered [ 7174.181800] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 7175.834666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7175.928337] alg: No test for adler32 (adler32-zlib) [ 7177.023191] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 7177.368568] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 7179.183118] Key type lgssc registered [ 7180.725639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7195.164330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7195.188440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.800636] Lustre: *** cfs_fail_loc=135, val=0*** [ 7196.928848] LustreError: 181572:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 7196.937888] LustreError: 181572:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.121@tcp: cfg command failed: rc = -2 [ 7196.947493] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7196.947493] [ 7196.967235] LustreError: MGC192.168.203.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7196.984531] LustreError: 181533:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7196.994395] LustreError: 181533:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7197.000901] LustreError: 181533:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7197.006136] LustreError: 181533:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7197.048932] Lustre: server umount lustre-MDT0000 complete [ 7197.051682] LustreError: 181533:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7207.005265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.659051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.392664] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7218.154713] LustreError: 181943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.181876] LustreError: 181943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7223.280148] LustreError: 181944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.385181] LustreError: 181943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7230.669477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7231.608722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7239.287622] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7250.945333] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7258.393891] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7262.180091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.184641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.196498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.209414] Lustre: Skipped 1 previous similar message [ 7262.230742] Lustre: Skipped 1 previous similar message [ 7267.304401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.312339] Lustre: Skipped 1 previous similar message [ 7268.585817] Lustre: server umount lustre-MDT0000 complete [ 7272.418598] LustreError: 181943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.457830] LustreError: 181943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7273.314156] LustreError: 181924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939433 with bad export cookie 10824997589885623940 [ 7273.320847] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.336968] LustreError: 181924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7274.079167] Lustre: server umount lustre-MDT0001 complete [ 7288.004674] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 7292.006124] Key type lgssc unregistered [ 7292.456381] LNet: 184273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7292.461351] LNetError: 184273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7292.498162] LNet: Removed LNI 192.168.203.121@tcp [ 7293.953100] Key type .llcrypt unregistered [ 7293.955662] Key type ._llcrypt unregistered [ 7322.214526] Key type ._llcrypt registered [ 7322.219995] Key type .llcrypt registered [ 7322.415382] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 7322.931362] Key type .llcrypt unregistered [ 7322.934441] Key type ._llcrypt unregistered [ 7349.255178] Key type ._llcrypt registered [ 7349.258784] Key type .llcrypt registered [ 7349.388857] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 7351.254614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7351.458546] alg: No test for adler32 (adler32-zlib) [ 7353.066726] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 7353.354559] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 7354.983732] Key type lgssc registered [ 7357.580466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.926397] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7368.939543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.543878] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7389.665587] LustreError: 186428:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.121@tcp: failed processing log, type 4: rc = -110 [ 7416.479235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7426.569559] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7437.342439] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.647513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.194212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.247845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7453.696483] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7455.733670] LustreError: 187467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7455.759753] LustreError: 187467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7460.852705] LustreError: 187466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.873284] LustreError: 187466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7465.956756] LustreError: 187793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7466.000060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.665885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.460713] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7483.750172] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.320808] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.176873] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7507.430503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7507.440700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.468077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7512.824933] Lustre: server umount lustre-OST0000 complete [ 7518.177727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.200423] Lustre: Skipped 1 previous similar message [ 7518.211316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7518.214178] Lustre: Skipped 1 previous similar message [ 7523.296654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.309476] Lustre: Skipped 1 previous similar message [ 7524.518893] Lustre: server umount lustre-MDT0000 complete [ 7527.402197] LustreError: 187793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7527.432672] LustreError: 187793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7529.470184] LustreError: 186435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939689 with bad export cookie 5803905755806475210 [ 7529.472126] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.487214] LustreError: 186435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7530.428934] Lustre: server umount lustre-MDT0001 complete [ 7547.251545] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 7551.488595] Key type lgssc unregistered [ 7551.904081] LNet: 190351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7551.910301] LNetError: 190351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7551.942698] LNet: Removed LNI 192.168.203.121@tcp [ 7552.974164] Key type .llcrypt unregistered [ 7552.979553] Key type ._llcrypt unregistered [ 7585.689512] Key type ._llcrypt registered [ 7585.690952] Key type .llcrypt registered [ 7585.897407] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 7587.640103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7587.892255] alg: No test for adler32 (adler32-zlib) [ 7589.042826] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 7589.413349] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 7591.199167] Key type lgssc registered [ 7593.028688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7604.790536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7604.821133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7606.785113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.279577] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7616.998462] LustreError: 191767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.029210] LustreError: 191767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7622.119495] LustreError: 191768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.370353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.152490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7632.552132] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7644.462917] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7652.407870] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7661.237047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7661.765211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7670.572217] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7677.203731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 7680.941723] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.040165] Lustre: Failing over lustre-MDT0000 [ 7688.836454] Lustre: server umount lustre-MDT0000 complete [ 7689.072134] LustreError: 194205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.770989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7692.777756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.800812] LustreError: 191763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.900026] Lustre: Skipped 2 previous similar messages [ 7697.890211] LustreError: 191767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.977085] LustreError: 191767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7708.139735] LustreError: 191767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.155103] LustreError: 191767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7709.024622] Lustre: 191081:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785939852/real 1785939852] req@000000001607b32c x1872693558278656/t0(0) o400->MGC192.168.203.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785939868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7709.079638] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7713.415950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.376985] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2dc937aee38df69e [ 7718.400147] Lustre: MGC192.168.203.121@tcp: Connection restored to 0@lo (at 0@lo) [ 7718.690865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.752687] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7720.751545] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7724.023362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7724.042286] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7724.110070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 7724.852650] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7730.639507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 7737.829400] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 7748.082799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7748.102637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7748.151389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7749.606306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7749.610194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.954496] Lustre: server umount lustre-OST0000 complete [ 7759.842503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7759.851489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7759.892105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7759.925957] Lustre: Skipped 2 previous similar messages [ 7764.970531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7765.382092] Lustre: server umount lustre-MDT0000 complete [ 7770.081248] LustreError: 194205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.143399] LustreError: 194205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 7771.086792] LustreError: 191749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785939930 with bad export cookie 3299229426307167902 [ 7771.103403] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.108862] LustreError: 191749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7771.644868] Lustre: server umount lustre-MDT0001 complete [ 7791.417957] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 7796.896876] Key type lgssc unregistered [ 7797.648922] LNet: 196632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7797.686353] LNetError: 196632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7797.758204] LNet: Removed LNI 192.168.203.121@tcp [ 7800.049309] Key type .llcrypt unregistered [ 7800.059237] Key type ._llcrypt unregistered [ 7840.467532] Key type ._llcrypt registered [ 7840.470569] Key type .llcrypt registered [ 7840.650022] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 7842.767415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7842.814971] alg: No test for adler32 (adler32-zlib) [ 7844.130266] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 7844.534122] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 7846.303229] Key type lgssc registered [ 7847.717764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7857.406490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7857.441518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.010507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.992322] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7868.387735] LustreError: 198050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.433612] LustreError: 198050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7873.516051] LustreError: 198051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.364272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.631230] LustreError: 198050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.349050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7886.995825] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7899.391077] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7907.420056] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.670354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.159299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7926.248854] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 7928.546839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 7937.506993] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7942.003426] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7948.835490] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 7967.373594] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7984.897508] Lustre: server umount lustre-OST0000 complete [ 7990.251891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.256210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.276066] Lustre: Skipped 1 previous similar message [ 7990.293820] Lustre: Skipped 1 previous similar message [ 7995.377870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7995.393929] Lustre: Skipped 1 previous similar message [ 7999.629929] Lustre: server umount lustre-MDT0000 complete [ 8000.488904] Lustre: 197359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785940144/real 1785940144] req@000000008e18d260 x1872693825688576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785940160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8000.489551] LustreError: 198050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8000.526870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8000.575147] LustreError: 198050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8001.260201] Lustre: 197361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785940144/real 1785940144] req@00000000b1fd0c1c x1872693825688832/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785940160 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8005.600872] LustreError: 198922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.638567] LustreError: 198922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8005.950564] LustreError: 198030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940165 with bad export cookie 17325068792344694839 [ 8005.964792] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8005.971946] LustreError: 198030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8006.834985] Lustre: server umount lustre-MDT0001 complete [ 8027.127439] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 8030.661165] Key type lgssc unregistered [ 8031.070220] LNet: 201966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8031.080346] LNetError: 201966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8031.138170] LNet: Removed LNI 192.168.203.121@tcp [ 8032.386060] Key type .llcrypt unregistered [ 8032.387518] Key type ._llcrypt unregistered [ 8063.273556] Key type ._llcrypt registered [ 8063.282733] Key type .llcrypt registered [ 8063.395881] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 8065.316721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8065.405949] alg: No test for adler32 (adler32-zlib) [ 8066.564428] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 8066.873485] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 8068.655726] Key type lgssc registered [ 8070.507890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8082.098290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8082.120833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.665178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.023687] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8094.176731] LustreError: 203237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8094.200586] LustreError: 203237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8099.297510] LustreError: 203238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.580764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.279579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8107.430725] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8118.588347] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8130.533908] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8142.918706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8143.430862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.524712] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8153.764158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8165.859753] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.909241] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8188.623334] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8197.323833] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8209.088390] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8220.641073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8220.651370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.662473] Lustre: Skipped 1 previous similar message [ 8220.664101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8220.682678] Lustre: Skipped 1 previous similar message [ 8224.324321] Lustre: server umount lustre-OST0000 complete [ 8230.880921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.902411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8230.904926] Lustre: Skipped 1 previous similar message [ 8235.667750] Lustre: server umount lustre-MDT0000 complete [ 8236.005640] LustreError: 204929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.051048] LustreError: 204929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8241.028111] LustreError: 205924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940400 with bad export cookie 16678983856518875237 [ 8241.032240] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8241.046453] LustreError: 205924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8241.444389] Lustre: server umount lustre-MDT0001 complete [ 8259.888965] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 8264.063707] Key type lgssc unregistered [ 8264.488127] LNet: 208031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8264.514521] LNetError: 208031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8264.535365] LNet: Removed LNI 192.168.203.121@tcp [ 8266.158652] Key type .llcrypt unregistered [ 8266.164268] Key type ._llcrypt unregistered [ 8299.242258] Key type ._llcrypt registered [ 8299.244385] Key type .llcrypt registered [ 8299.308525] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 8300.686428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8300.740578] alg: No test for adler32 (adler32-zlib) [ 8302.010149] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 8302.548340] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 8304.447186] Key type lgssc registered [ 8306.739958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8318.692771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8318.708483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8320.434672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.363249] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8330.723478] LustreError: 209450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.745618] LustreError: 209450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8335.843429] LustreError: 209449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8338.214192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.833938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8344.490328] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8354.111936] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8363.133765] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8376.342464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.022142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8385.555903] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8390.412688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8395.505912] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8420.320061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8420.334616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8420.366205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.354358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8421.360510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.382086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.443182] Lustre: Skipped 1 previous similar message [ 8424.101807] Lustre: server umount lustre-OST0000 complete [ 8431.072165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8431.077148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8431.090076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8436.150927] Lustre: server umount lustre-MDT0000 complete [ 8436.707434] LustreError: 209450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8436.727575] LustreError: 209450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8441.453213] LustreError: 209429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940601 with bad export cookie 8183142659881162640 [ 8441.461297] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8441.466466] LustreError: 209429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8442.223803] Lustre: server umount lustre-MDT0001 complete [ 8457.435748] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 8461.440645] Key type lgssc unregistered [ 8461.998589] LNet: 213138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8462.001915] LNetError: 213138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8463.014978] LNet: Removed LNI 192.168.203.121@tcp [ 8464.364636] Key type .llcrypt unregistered [ 8464.368441] Key type ._llcrypt unregistered [ 8489.240323] Key type ._llcrypt registered [ 8489.241956] Key type .llcrypt registered [ 8489.401749] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 8490.935915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8491.062497] alg: No test for adler32 (adler32-zlib) [ 8492.155578] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 8492.364731] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 8494.215365] Key type lgssc registered [ 8495.207100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8507.239650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8507.319308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.841134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.325209] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8519.138305] LustreError: 214408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8519.167146] LustreError: 214408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8524.260481] LustreError: 214407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.900618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.608425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8535.960672] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8547.370393] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8555.742504] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8566.115706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8566.567237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.534357] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8578.871958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8584.422598] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8604.644940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8604.655334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.664206] LustreError: Skipped 1 previous similar message [ 8604.685048] Lustre: Skipped 1 previous similar message [ 8604.704596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8604.717126] Lustre: Skipped 1 previous similar message [ 8609.761449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8609.770368] Lustre: Skipped 1 previous similar message [ 8611.034761] Lustre: server umount lustre-OST0000 complete [ 8620.003688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.021341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.030531] Lustre: Skipped 1 previous similar message [ 8624.007352] Lustre: server umount lustre-MDT0000 complete [ 8625.128431] LustreError: 214407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8625.160053] LustreError: 214407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8628.662403] LustreError: 214389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785940788 with bad export cookie 4871729806512211016 [ 8628.678077] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.983713] Lustre: server umount lustre-MDT0001 complete [ 8643.783948] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 8647.568804] Key type lgssc unregistered [ 8647.957559] LNet: 218098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8647.966551] LNetError: 218098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8647.994352] LNet: Removed LNI 192.168.203.121@tcp [ 8649.048658] Key type .llcrypt unregistered [ 8649.050518] Key type ._llcrypt unregistered [ 8673.597727] Key type ._llcrypt registered [ 8673.599063] Key type .llcrypt registered [ 8673.710249] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 8675.335147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8675.386273] alg: No test for adler32 (adler32-zlib) [ 8677.229730] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 8677.640863] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 8679.399139] Key type lgssc registered [ 8681.674784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8697.213127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8697.301468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.312554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.139758] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8709.600168] LustreError: 219365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.630469] LustreError: 219365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8714.737603] LustreError: 219366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8719.846797] LustreError: 219365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8722.075682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.739860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8730.757418] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8740.374300] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8747.991354] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.090489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.701199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.229890] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8771.002147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8774.639491] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8786.781294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8786.962697] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8787.416421] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8796.276628] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 8805.620471] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8807.999217] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8820.396657] Lustre: Permanently deactivating lustre-OST0001 [ 8820.457613] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8826.486952] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8837.414452] Lustre: Permanently reactivating lustre-OST0001 [ 8837.441976] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8837.445989] Lustre: Skipped 2 previous similar messages [ 8843.361036] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.380159] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8843.396414] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8843.414731] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8843.451341] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8848.864407] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8848.867269] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.883832] LustreError: Skipped 1 previous similar message [ 8848.892467] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8848.913878] Lustre: Skipped 2 previous similar messages [ 8848.948804] Lustre: Skipped 1 previous similar message [ 8850.627923] Lustre: server umount lustre-OST0001 complete [ 8850.914015] LustreError: 222927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.960920] LustreError: 222927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8856.033126] LustreError: 221191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8856.041980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.054982] LustreError: 221191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8856.093284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8856.106052] Lustre: Skipped 1 previous similar message [ 8861.153444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8861.161433] Lustre: Skipped 1 previous similar message [ 8866.278470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8866.291294] LustreError: 222927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.299214] Lustre: Skipped 1 previous similar message [ 8866.321617] LustreError: 222927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8870.367179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8871.003140] Lustre: server umount lustre-OST0000 complete [ 8881.632075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8881.635075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.642774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8881.651827] Lustre: Skipped 2 previous similar messages [ 8881.669809] Lustre: Skipped 1 previous similar message [ 8883.668025] Lustre: server umount lustre-MDT0000 complete [ 8886.754761] LustreError: 219365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.794661] LustreError: 219365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8889.770259] LustreError: 219344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941049 with bad export cookie 18052402109682402766 [ 8889.785000] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.793538] LustreError: 219344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8891.172088] Lustre: server umount lustre-MDT0001 complete [ 8906.329273] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 8911.017850] Key type lgssc unregistered [ 8911.583593] LNet: 224425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8911.602858] LNetError: 224425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8912.684464] LNet: Removed LNI 192.168.203.121@tcp [ 8914.171510] Key type .llcrypt unregistered [ 8914.174894] Key type ._llcrypt unregistered [ 8969.667451] Key type ._llcrypt registered [ 8969.676476] Key type .llcrypt registered [ 8969.817536] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 8972.360279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8972.487848] alg: No test for adler32 (adler32-zlib) [ 8973.956319] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 8974.762279] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 8976.487161] Key type lgssc registered [ 8979.241664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8991.304282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8991.323071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8992.739694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8992.771307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8992.926903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8998.104851] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 9009.796282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.946992] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9010.033177] Lustre: 226634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9010.343272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9015.677292] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 9026.046528] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9033.537081] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9044.888538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9045.163474] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9045.507737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9055.429038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 9056.243704] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 9066.823161] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9070.371558] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9076.713119] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9081.934959] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9088.502591] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9105.756511] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9105.768164] Lustre: Skipped 1 previous similar message [ 9127.475858] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9127.493979] Lustre: Skipped 2 previous similar messages [ 9161.942630] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9161.952566] Lustre: Skipped 3 previous similar messages [ 9233.278205] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 9233.286954] Lustre: Skipped 8 previous similar messages [ 9245.158575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9245.176205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9245.178219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9245.195533] Lustre: Skipped 1 previous similar message [ 9249.542744] Lustre: server umount lustre-OST0000 complete [ 9255.395829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.410222] Lustre: Skipped 1 previous similar message [ 9255.416612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9255.436535] Lustre: Skipped 1 previous similar message [ 9260.406986] Lustre: server umount lustre-MDT0000 complete [ 9264.611667] LustreError: 226627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941424 with bad export cookie 12094340414568803344 [ 9264.623395] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9264.625493] LustreError: 226627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9265.155291] Lustre: server umount lustre-MDT0001 complete [ 9279.703838] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 9283.907857] Key type lgssc unregistered [ 9284.378533] LNet: 231462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9284.387549] LNetError: 231462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9284.413878] LNet: Removed LNI 192.168.203.121@tcp [ 9286.108443] Key type .llcrypt unregistered [ 9286.115051] Key type ._llcrypt unregistered [ 9315.461865] Key type ._llcrypt registered [ 9315.466600] Key type .llcrypt registered [ 9315.571888] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 9316.707073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9316.762585] alg: No test for adler32 (adler32-zlib) [ 9317.787322] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 9318.212336] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 9319.967156] Key type lgssc registered [ 9321.553378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9332.349234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9332.370146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.118435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.183803] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 9343.477887] LustreError: 232877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.530689] LustreError: 232877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9348.584891] LustreError: 232876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9351.983583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9352.490355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9358.499079] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 9369.000359] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9375.014397] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9384.930290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9385.400708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9395.894588] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_default_debug -1 all [ 9403.950468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 9409.190562] Lustre: DEBUG MARKER: oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9433.576000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9433.583083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.599883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.596608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9434.600917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9434.617358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.632667] Lustre: server umount lustre-OST0000 complete [ 9444.832725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.834343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9444.861404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.861412] Lustre: Skipped 1 previous similar message [ 9449.952599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9449.962975] Lustre: Skipped 2 previous similar messages [ 9451.186631] Lustre: server umount lustre-MDT0000 complete [ 9455.085204] LustreError: 232877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.120265] LustreError: 232877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9457.141592] LustreError: 232856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785941616 with bad export cookie 2533519655173076688 [ 9457.143146] LustreError: MGC192.168.203.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9457.160755] LustreError: 232856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9458.046930] Lustre: server umount lustre-MDT0001 complete [ 9484.440556] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 9488.606679] Key type lgssc unregistered [ 9489.041654] LNet: 236594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9489.050227] LNetError: 236594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9489.074666] LNet: Removed LNI 192.168.203.121@tcp [ 9490.432135] Key type .llcrypt unregistered [ 9490.434951] Key type ._llcrypt unregistered [ 9519.730923] Key type ._llcrypt registered [ 9519.732577] Key type .llcrypt registered [ 9519.932382] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 9521.598694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9521.650324] alg: No test for adler32 (adler32-zlib) [ 9522.796926] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 9523.156368] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 9524.952200] Key type lgssc registered [ 9527.273933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9801.871740] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 9805.537675] Key type lgssc unregistered [ 9805.864596] LNet: 238625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9805.871568] LNetError: 238625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9805.890365] LNet: Removed LNI 192.168.203.121@tcp [ 9807.247773] Key type .llcrypt unregistered [ 9807.251964] Key type ._llcrypt unregistered [ 9842.969976] Key type ._llcrypt registered [ 9842.975522] Key type .llcrypt registered [ 9843.133995] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing set_hostid [ 9864.421956] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing load_modules_local [ 9865.859234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9866.002079] alg: No test for adler32 (adler32-zlib) [ 9867.419695] Lustre: Lustre: Build Version: 2.17.56_50_g82e980f [ 9867.791884] LNet: Added LNI 192.168.203.121@tcp [8/256/0/180] [ 9869.487419] Key type lgssc registered [ 9871.350033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9911.477621] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9571 sec ========== 11:01:08 (1785942068) [ 9913.787826] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:01:11 (1785942071) === [ 9918.283790] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:01:15 (1785942075) === [ 9969.793871] Lustre: DEBUG MARKER: oleg321-server.virtnet: executing unload_modules_local [ 9973.241948] Key type lgssc unregistered [ 9973.684539] LNet: 243884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9973.691669] LNetError: 243884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9973.715714] LNet: Removed LNI 192.168.203.121@tcp [ 9974.795392] Key type .llcrypt unregistered [ 9974.799810] Key type ._llcrypt unregistered