[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487201410 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007414] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011076] Yama: becoming mindful. [ 0.013023] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.023006] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028708] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030080] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032116] *** VALIDATE tmpfs *** [ 0.034361] *** VALIDATE proc *** [ 0.036053] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.038270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042012] Speculative Store Bypass: Vulnerable [ 0.045145] debug: unmapping init [mem 0xffffffffb2059000-0xffffffffb2060fff] [ 0.047192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048524] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053019] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056321] rcu: Hierarchical SRCU implementation. [ 0.058495] smp: Bringing up secondary CPUs ... [ 0.059632] x86: Booting SMP configuration: [ 0.060030] .... node #0, CPUs: #1 #2 #3 [ 0.063190] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143147] node 0 deferred pages initialised in 74ms [ 0.148134] devtmpfs: initialized [ 0.149277] x86/mm: Memory block size: 128MB [ 0.152021] gcov: version magic: 0x41383552 [ 0.154337] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160400] pinctrl core: initialized pinctrl subsystem [ 0.163220] [ 0.164009] ************************************************************* [ 0.167015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170015] ** ** [ 0.173012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175012] ** ** [ 0.177016] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.182014] ** your system. ** [ 0.185016] ** ** [ 0.187014] ** If you see this message and you are not debugging the ** [ 0.190013] ** kernel, report this immediately to your vendor! ** [ 0.192014] ** ** [ 0.195015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198014] ************************************************************* [ 0.200483] NET: Registered protocol family 16 [ 0.202527] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.206066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211435] cpuidle: using governor menu [ 0.213822] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216481] PCI: Using configuration type 1 for base access [ 0.218137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231097] cryptd: max_cpu_qlen set to 1000 [ 0.235338] ACPI: Added _OSI(Module Device) [ 0.237020] ACPI: Added _OSI(Processor Device) [ 0.238014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244492] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251672] ACPI: Interpreter enabled [ 0.254083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256013] ACPI: Using IOAPIC for interrupt routing [ 0.258119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261635] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274812] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.278047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.281018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291000] acpiphp: Slot [2] registered [ 0.292080] acpiphp: Slot [5] registered [ 0.294106] acpiphp: Slot [6] registered [ 0.295135] acpiphp: Slot [7] registered [ 0.297122] acpiphp: Slot [8] registered [ 0.299102] acpiphp: Slot [9] registered [ 0.301143] acpiphp: Slot [10] registered [ 0.303140] acpiphp: Slot [3] registered [ 0.304105] acpiphp: Slot [4] registered [ 0.305075] acpiphp: Slot [11] registered [ 0.307214] acpiphp: Slot [12] registered [ 0.309142] acpiphp: Slot [13] registered [ 0.310183] acpiphp: Slot [14] registered [ 0.312098] acpiphp: Slot [15] registered [ 0.314092] acpiphp: Slot [16] registered [ 0.315105] acpiphp: Slot [17] registered [ 0.317113] acpiphp: Slot [18] registered [ 0.319179] acpiphp: Slot [19] registered [ 0.320109] acpiphp: Slot [20] registered [ 0.322109] acpiphp: Slot [21] registered [ 0.324097] acpiphp: Slot [22] registered [ 0.325086] acpiphp: Slot [23] registered [ 0.326103] acpiphp: Slot [24] registered [ 0.328138] acpiphp: Slot [25] registered [ 0.330119] acpiphp: Slot [26] registered [ 0.332138] acpiphp: Slot [27] registered [ 0.333122] acpiphp: Slot [28] registered [ 0.335114] acpiphp: Slot [29] registered [ 0.337132] acpiphp: Slot [30] registered [ 0.338115] acpiphp: Slot [31] registered [ 0.340063] PCI host bridge to bus 0000:00 [ 0.342020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.347023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.350057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.355025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357279] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361187] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364473] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380914] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.386019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.389019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.393166] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395802] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.403876] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.425018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.431017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.436463] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.456018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.500743] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.507016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.533016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.542486] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.578068] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.584014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.609017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.620393] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.626015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.661834] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.671015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.682017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.715016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.729125] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.732380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735405] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.738383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.741318] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746125] iommu: Default domain type: Passthrough [ 0.749475] SCSI subsystem initialized [ 0.751136] ACPI: bus type USB registered [ 0.752156] usbcore: registered new interface driver usbfs [ 0.754161] usbcore: registered new interface driver hub [ 0.757087] usbcore: registered new device driver usb [ 0.759160] pps_core: LinuxPPS API ver. 1 registered [ 0.761013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.764064] PTP clock support registered [ 0.766143] EDAC MC: Ver: 3.0.0 [ 0.768626] PCI: Using ACPI for IRQ routing [ 0.770900] NetLabel: Initializing [ 0.772015] NetLabel: domain hash size = 128 [ 0.774012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.776090] NetLabel: unlabeled traffic allowed by default [ 0.779112] vgaarb: loaded [ 0.780283] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.783015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.788349] clocksource: Switched to clocksource kvm-clock [ 0.899702] VFS: Disk quotas dquot_6.6.0 [ 0.901205] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.904608] *** VALIDATE ramfs *** [ 0.905724] *** VALIDATE hugetlbfs *** [ 0.907359] pnp: PnP ACPI init [ 0.910307] pnp: PnP ACPI: found 6 devices [ 0.933380] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.936839] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.939175] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.941364] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.943902] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.946185] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.948736] NET: Registered protocol family 2 [ 0.951644] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.957961] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.961783] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.967962] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.972132] TCP: Hash tables configured (established 65536 bind 65536) [ 0.975605] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.979198] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.982262] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985935] NET: Registered protocol family 1 [ 0.989850] RPC: Registered named UNIX socket transport module. [ 0.992684] RPC: Registered udp transport module. [ 0.994421] RPC: Registered tcp transport module. [ 0.996382] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.998978] NET: Registered protocol family 44 [ 1.000859] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.003897] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.006354] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.009137] PCI: CLS 0 bytes, default 64 [ 1.011052] Unpacking initramfs... [ 2.438248] debug: unmapping init [mem 0xffff9349bcc54000-0xffff9349bffbffff] [ 2.443652] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.447324] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.452264] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.956147] Initialise system trusted keyrings [ 2.958573] Key type blacklist registered [ 2.960324] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.970461] zbud: loaded [ 2.973663] *** VALIDATE nfs *** [ 2.974514] *** VALIDATE nfs4 *** [ 2.975942] pstore: using deflate compression [ 2.979238] Platform Keyring initialized [ 3.082387] NET: Registered protocol family 38 [ 3.084079] Key type asymmetric registered [ 3.085475] Asymmetric key parser 'x509' registered [ 3.087440] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.090492] io scheduler mq-deadline registered [ 3.092315] io scheduler kyber registered [ 3.094260] io scheduler bfq registered [ 3.096481] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.099944] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.103220] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.106556] ACPI: Power Button [PWRF] [ 3.111780] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.117957] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.130154] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.136756] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.149989] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.177317] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.209448] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.214443] Non-volatile memory driver v1.3 [ 3.216339] Linux agpgart interface v0.103 [ 3.248564] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.251156] vda: detected capacity change from 0 to 75108352 [ 3.267432] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.271209] vdb: detected capacity change from 0 to 1073741824 [ 3.287939] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.290964] vdc: detected capacity change from 0 to 2621440000 [ 3.310635] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.313315] vdd: detected capacity change from 0 to 2621440000 [ 3.327041] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.330020] vde: detected capacity change from 0 to 4294967296 [ 3.344653] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.347928] vdf: detected capacity change from 0 to 4294967296 [ 3.353499] libphy: Fixed MDIO Bus: probed [ 3.358365] usbcore: registered new interface driver usbserial_generic [ 3.361202] usbserial: USB Serial support registered for generic [ 3.363870] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.368094] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.370142] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.372843] mousedev: PS/2 mouse device common for all mice [ 3.376046] rtc_cmos 00:05: RTC can wake from S4 [ 3.378698] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.382114] rtc_cmos 00:05: registered as rtc0 [ 3.384344] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.385015] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.390331] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.391836] intel_pstate: CPU model not supported [ 3.399193] hid: raw HID events driver (C) Jiri Kosina [ 3.401592] usbcore: registered new interface driver usbhid [ 3.403449] usbhid: USB HID core driver [ 3.405248] drop_monitor: Initializing network drop monitor service [ 3.407686] Initializing XFRM netlink socket [ 3.409924] NET: Registered protocol family 10 [ 3.414074] Segment Routing with IPv6 [ 3.415810] NET: Registered protocol family 17 [ 3.418129] mpls_gso: MPLS GSO support [ 3.424447] RAS: Correctable Errors collector initialized. [ 3.426555] AVX version of gcm_enc/dec engaged. [ 3.428191] AES CTR mode by8 optimization enabled [ 3.504868] sched_clock: Marking stable (3504844679, 0)->(4430074810, -925230131) [ 3.508697] registered taskstats version 1 [ 3.511796] Loading compiled-in X.509 certificates [ 3.514084] zswap: loaded using pool lzo/zbud [ 3.537344] Key type big_key registered [ 3.550205] Key type encrypted registered [ 3.551683] ima: No TPM chip found, activating TPM-bypass! [ 3.553792] ima: Allocated hash algorithm: sha1 [ 3.555703] ima: No architecture policies found [ 3.557574] evm: Initialising EVM extended attributes: [ 3.559311] evm: security.selinux [ 3.560286] evm: security.ima [ 3.561018] evm: security.capability [ 3.561967] evm: HMAC attrs: 0x1 [ 3.564103] rtc_cmos 00:05: setting system clock to 2026-09-08 01:38:37 UTC (1788831517) [ 3.570715] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.574629] debug: unmapping init [mem 0xffffffffb1d82000-0xffffffffb2058fff] [ 3.583086] Write protecting the kernel read-only data: 28672k [ 3.587100] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.589867] debug: unmapping init [mem 0xffffffffb0d14000-0xffffffffb0dfffff] [ 3.625315] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.634197] systemd[1]: Detected virtualization kvm. [ 3.636081] systemd[1]: Detected architecture x86-64. [ 3.638119] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.663899] systemd[1]: No hostname configured. [ 3.665364] systemd[1]: Set hostname to . [ 3.667604] random: systemd: uninitialized urandom read (16 bytes read) [ 3.670315] systemd[1]: Initializing machine ID from random generator. [ 3.727119] random: ln: uninitialized urandom read (6 bytes read) [ 3.810863] random: systemd: uninitialized urandom read (16 bytes read) [ 3.814268] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.821635] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.826525] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.499710] device-mapper: uevent: version 1.0.3 [ 4.502603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.184097] random: fast init done [ 5.199110] virtio_net virtio0 ens2: renamed from eth0 [ 5.252706] scsi host0: ata_piix [ 5.281336] scsi host1: ata_piix [ 5.284315] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.286992] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.018890] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.065989] random: crng init done [ 10.067457] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.537640] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.646668] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.944337] SELinux: Disabled at runtime. [ 12.007213] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.015905] systemd[1]: Detected virtualization kvm. [ 12.017986] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.526521] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.530362] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.536262] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.540501] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.543500] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.551455] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.556453] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ 12.634855] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.978404] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.265045] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.286179] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.331470] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.348987] EDAC sbridge: Ver: 1.1.2 [ 15.130567] Key type dns_resolver registered [ 15.442754] NFS: Registering the id_resolver key type [ 15.444927] Key type id_resolver registered [ 15.446771] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg418-server login: [ 42.147372] libcfs: loading out-of-tree module taints kernel. [ 42.170598] Key type ._llcrypt registered [ 42.172329] Key type .llcrypt registered [ 42.229913] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_hostid [ 61.848525] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 63.304612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.324245] alg: No test for adler32 (adler32-zlib) [ 64.627607] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 65.429249] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 67.152531] Key type lgssc registered [ 69.127791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.645081] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.178075] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 152.307627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.338211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.537402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.617880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.811603] Lustre: lustre-MDT0000: new disk, initializing [ 153.991279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.011356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.694458] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.684182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.849607] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 174.905155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 174.916877] Lustre: Skipped 1 previous similar message [ 175.032301] Lustre: lustre-MDT0001: new disk, initializing [ 175.091785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.112573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.125805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.657517] hrtimer: interrupt took 4506794 ns [ 180.072791] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.610706] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.577504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.970386] Lustre: lustre-OST0000: new disk, initializing [ 197.977541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.984641] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.079480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.370186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.390478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.514654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.924841] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.057962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.322279] Lustre: lustre-OST0001: new disk, initializing [ 221.326519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.341123] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.492985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.581388] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.071760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.080656] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.179216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.189383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.648136] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 254.603564] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.541846] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing yml_node [ 267.321778] Lustre: DEBUG MARKER: Client: 2.17.57.110 [ 269.868414] Lustre: DEBUG MARKER: MDS: 2.17.57.110 [ 272.563722] Lustre: DEBUG MARKER: OSS: 2.17.57.110 [ 274.320790] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 21:43:05 EDT 2026 [ 294.809541] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 296.533469] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 317.421126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.442837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.453610] Lustre: Skipped 2 previous similar messages [ 317.484190] Lustre: Skipped 3 previous similar messages [ 320.038848] Lustre: server umount lustre-MDT0000 complete [ 324.236355] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831838 with bad export cookie 12360471925907964102 [ 324.244580] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.247121] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 327.649347] LustreError: 6507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.650787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.675299] LustreError: 6507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 327.696511] Lustre: Skipped 1 previous similar message [ 327.697658] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.701762] LustreError: 9938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.701960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.727361] Lustre: Skipped 2 previous similar messages [ 330.639532] Lustre: server umount lustre-MDT0001 complete [ 335.391233] Lustre: server umount lustre-OST0000 complete [ 339.483469] Lustre: server umount lustre-OST0001 complete [ 346.893870] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_hostid [ 358.625444] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 397.227798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.610401] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 397.663258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 397.768839] Lustre: lustre-MDT0000: new disk, initializing [ 397.844274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.856036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 402.802272] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 413.527410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.707227] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.798600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.802504] Lustre: Skipped 1 previous similar message [ 413.897177] Lustre: lustre-MDT0001: new disk, initializing [ 413.992896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.039740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.047444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.991035] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 429.815941] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 436.345284] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 443.317717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.607674] Lustre: lustre-OST0000: new disk, initializing [ 443.611284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.625472] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.727077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.287178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 445.294899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 445.479694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 450.914922] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 462.928493] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 468.601834] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 468.850378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.161727] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 474.396742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.259107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.260986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.272449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.313826] Lustre: Skipped 2 previous similar messages [ 482.777661] Lustre: server umount lustre-OST0000 complete [ 486.373822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.383080] Lustre: Skipped 1 previous similar message [ 486.384899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.402039] Lustre: Skipped 2 previous similar messages [ 492.615737] Lustre: server umount lustre-MDT0000 complete [ 496.345812] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832010 with bad export cookie 12360471925907965635 [ 496.350731] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.357717] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 496.761914] Lustre: server umount lustre-MDT0001 complete [ 500.493222] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:46:51 (1788832011) [ 509.474777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.931917] LustreError: 19304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.948625] LustreError: 19304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 510.013992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.049454] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.823514] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 519.140645] LustreError: 19304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.908722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.492418] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 538.879702] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 544.684638] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 552.015657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.539367] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 569.852930] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 582.116320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.124213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.135617] LustreError: Skipped 2 previous similar messages [ 582.138896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 582.138907] Lustre: Skipped 2 previous similar messages [ 582.215615] Lustre: Skipped 1 previous similar message [ 587.654523] Lustre: server umount lustre-OST0000 complete [ 592.867546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 592.883530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.928832] Lustre: server umount lustre-MDT0000 complete [ 600.545970] LustreError: 20696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.569359] LustreError: 20696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 601.836357] LustreError: 20040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832115 with bad export cookie 12360471925907966517 [ 601.840831] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 601.858427] LustreError: 20040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 602.407157] Lustre: server umount lustre-MDT0001 complete [ 614.723236] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 617.713252] Key type lgssc unregistered [ 617.987962] LNet: 22836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.998347] LNetError: 22836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.027022] LNet: Removed LNI 192.168.204.118@tcp [ 619.294409] Key type .llcrypt unregistered [ 619.300905] Key type ._llcrypt unregistered [ 647.734947] Key type ._llcrypt registered [ 647.738742] Key type .llcrypt registered [ 647.854263] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 648.761963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.823671] alg: No test for adler32 (adler32-zlib) [ 650.110927] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 650.424320] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 652.080180] Key type lgssc registered [ 653.476728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 661.471135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 661.494637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.987826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.090387] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 672.228536] LustreError: 24248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.245407] LustreError: 24248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 675.913261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.241853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 679.574602] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 687.046575] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 693.569795] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.071936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 701.502315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.637839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 708.413304] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 716.524542] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 731.647968] Lustre: server umount lustre-OST0000 complete [ 737.761684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 737.775870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 737.797533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.277818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.301184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 738.306556] Lustre: Skipped 1 previous similar message [ 743.393616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.401731] Lustre: Skipped 1 previous similar message [ 745.573867] Lustre: server umount lustre-MDT0000 complete [ 748.526147] LustreError: 24974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 748.550698] LustreError: 24974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 749.536941] Lustre: 23556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832247/real 1788832247] req@0000000023781d53 x1875726475227520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788832263 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 749.537254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.564705] Lustre: 23556:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 749.604858] LustreError: 24244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.622769] LustreError: 24244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 749.839963] LustreError: 24228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832263 with bad export cookie 1270365683199856451 [ 749.841369] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.856664] LustreError: 24228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 750.189540] Lustre: server umount lustre-MDT0001 complete [ 762.322764] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 766.031838] Key type lgssc unregistered [ 766.486844] LNet: 28225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.501148] LNetError: 28225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.523037] LNet: Removed LNI 192.168.204.118@tcp [ 767.397229] Key type .llcrypt unregistered [ 767.402084] Key type ._llcrypt unregistered [ 795.419750] Key type ._llcrypt registered [ 795.423584] Key type .llcrypt registered [ 795.508751] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 796.249244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.275821] alg: No test for adler32 (adler32-zlib) [ 797.327589] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 797.556219] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 799.240739] Key type lgssc registered [ 800.387168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 808.839400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.863416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.228790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.010833] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 820.707565] LustreError: 29638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.722156] LustreError: 29638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 825.839359] LustreError: 29637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 826.254539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.958920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 832.386861] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 840.455317] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 846.408749] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 862.940859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.670055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.570165] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 878.893704] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 882.130753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 892.387948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.392510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 892.393956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 892.403258] Lustre: Skipped 1 previous similar message [ 896.343338] Lustre: server umount lustre-OST0000 complete [ 902.635636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.641753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.652560] Lustre: Skipped 1 previous similar message [ 902.666125] Lustre: Skipped 2 previous similar messages [ 906.866653] Lustre: server umount lustre-MDT0000 complete [ 907.748158] LustreError: 29634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.765736] LustreError: 29634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 911.035827] LustreError: 30372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832424 with bad export cookie 16456622162019127788 [ 911.039158] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.051593] LustreError: 30372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 911.462492] Lustre: server umount lustre-MDT0001 complete [ 924.437429] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 928.011227] Key type lgssc unregistered [ 928.322852] LNet: 33517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.335130] LNetError: 33517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.358186] LNet: Removed LNI 192.168.204.118@tcp [ 929.285829] Key type .llcrypt unregistered [ 929.289209] Key type ._llcrypt unregistered [ 952.860791] Key type ._llcrypt registered [ 952.865105] Key type .llcrypt registered [ 953.041607] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 954.524967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.568151] alg: No test for adler32 (adler32-zlib) [ 955.607461] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 955.981795] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 957.688207] Key type lgssc registered [ 958.863953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 966.661643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 966.677597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.193353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.816450] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 978.408048] LustreError: 34925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.443213] LustreError: 34925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 981.281059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.668547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 986.813927] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 995.887339] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1002.841352] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1010.938808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1011.408625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.450571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1017.977972] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1025.820568] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1037.285991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.289171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.311689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.019602] Lustre: server umount lustre-OST0000 complete [ 1047.530096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.552830] Lustre: Skipped 1 previous similar message [ 1047.561571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.571832] Lustre: Skipped 2 previous similar messages [ 1052.641470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.649765] Lustre: Skipped 1 previous similar message [ 1057.762397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.320266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1060.507286] Lustre: server umount lustre-MDT0000 complete [ 1062.881471] LustreError: 37527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.906966] LustreError: 37527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1064.535917] LustreError: 34904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832578 with bad export cookie 11696119309306078088 [ 1064.536430] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.548918] LustreError: 34904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1064.885950] Lustre: server umount lustre-MDT0001 complete [ 1077.061601] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 1079.649218] Key type lgssc unregistered [ 1080.056506] LNet: 38461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.065147] LNetError: 38461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.076224] LNet: Removed LNI 192.168.204.118@tcp [ 1080.838098] Key type .llcrypt unregistered [ 1080.842383] Key type ._llcrypt unregistered [ 1110.022952] Key type ._llcrypt registered [ 1110.024438] Key type .llcrypt registered [ 1110.114435] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 1111.025908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.041642] alg: No test for adler32 (adler32-zlib) [ 1112.205736] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1112.666623] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 1114.408350] Key type lgssc registered [ 1116.017309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1124.636748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1124.653350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.960738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.854693] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1135.081547] LustreError: 39873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.109899] LustreError: 39873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1139.695649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.055527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1145.374218] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1154.346172] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1160.820767] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.003836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.450838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.488179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1175.783785] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1185.741978] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.935459] Lustre: lustre-OST0000: Not available for connect from 192.168.204.18@tcp (stopping) [ 1194.978973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1194.993370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1195.002807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1195.046495] Lustre: Skipped 1 previous similar message [ 1198.745570] Lustre: server umount lustre-OST0000 complete [ 1220.578411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.585856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.594710] Lustre: Skipped 1 previous similar message [ 1220.597805] Lustre: Skipped 1 previous similar message [ 1225.703248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.713606] Lustre: Skipped 2 previous similar messages [ 1226.793755] Lustre: server umount lustre-MDT0000 complete [ 1230.098387] LustreError: 40607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832744 with bad export cookie 12557591107198126567 [ 1230.099406] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.103963] LustreError: 40607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1236.389274] Lustre: server umount lustre-MDT0001 complete [ 1246.706314] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 1249.475526] Key type lgssc unregistered [ 1249.702910] LNet: 43579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.710825] LNetError: 43579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.730226] LNet: Removed LNI 192.168.204.118@tcp [ 1250.496439] Key type .llcrypt unregistered [ 1250.499188] Key type ._llcrypt unregistered [ 1276.737429] Key type ._llcrypt registered [ 1276.740812] Key type .llcrypt registered [ 1276.874340] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 1277.804033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1277.828413] alg: No test for adler32 (adler32-zlib) [ 1278.966408] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1279.216559] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 1280.912170] Key type lgssc registered [ 1282.159117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1290.385248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1290.407118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1291.922328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.590484] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1301.474151] LustreError: 44987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.502806] LustreError: 44987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1306.597640] LustreError: 44988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.631871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.165959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1312.640172] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1321.791327] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1328.499247] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1336.135749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.516448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.592056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1343.573445] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1352.925994] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1360.857643] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.18@tcp (stopping) [ 1361.785316] Lustre: server umount lustre-MDT0000 complete [ 1362.410133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.433853] LustreError: 46246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.463432] LustreError: 46246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1365.488654] LustreError: 44969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832879 with bad export cookie 2677839592982294824 [ 1365.494833] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.507909] LustreError: 44969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1365.939170] LustreError: 44983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.983919] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.18@tcp (stopping) [ 1367.521826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.541828] Lustre: Skipped 2 previous similar messages [ 1367.547176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1370.529278] LustreError: 44984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.529639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1370.544791] LustreError: 44984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1372.098423] Lustre: server umount lustre-MDT0001 complete [ 1401.312985] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1401.606105] Lustre: server umount lustre-OST0000 complete [ 1420.015663] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 1422.986768] Key type lgssc unregistered [ 1423.224848] LNet: 48862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.232064] LNetError: 48862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.243854] LNet: Removed LNI 192.168.204.118@tcp [ 1424.056651] Key type .llcrypt unregistered [ 1424.063955] Key type ._llcrypt unregistered [ 1451.648611] Key type ._llcrypt registered [ 1451.651519] Key type .llcrypt registered [ 1451.766441] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 1453.008455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1453.226783] alg: No test for adler32 (adler32-zlib) [ 1454.408875] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1454.685912] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 1456.344165] Key type lgssc registered [ 1457.930139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1467.417235] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1467.435426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1482.976387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1488.097069] LustreError: 50249:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 1514.785209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.390455] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1529.705903] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1697.657476] Lustre: server umount lustre-OST0000 complete [ 1715.278663] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 1718.127614] Key type lgssc unregistered [ 1718.446588] LNet: 51924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.457500] LNetError: 51924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.529894] LNet: Removed LNI 192.168.204.118@tcp [ 1720.356799] Key type .llcrypt unregistered [ 1720.361423] Key type ._llcrypt unregistered [ 1744.803876] Key type ._llcrypt registered [ 1744.806282] Key type .llcrypt registered [ 1744.908487] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 1745.953439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.103573] alg: No test for adler32 (adler32-zlib) [ 1747.210938] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1747.471325] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 1749.184169] Key type lgssc registered [ 1750.290506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.299902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.319137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.786456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.107344] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1769.957796] LustreError: 53338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.984313] LustreError: 53338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1772.750902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.143985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1777.458298] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1785.529694] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1791.214769] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1798.074562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1798.357307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.616862] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1804.556802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1812.170337] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1815.010731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.035275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.131022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.177993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.192308] Lustre: Skipped 1 previous similar message [ 1827.302893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1827.310609] Lustre: Skipped 1 previous similar message [ 1829.344737] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1829.512972] Lustre: server umount lustre-OST0000 complete [ 1834.465825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1834.472207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.506894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1837.537590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.840844] Lustre: server umount lustre-MDT0000 complete [ 1842.661805] LustreError: 53337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.699767] LustreError: 53337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1843.996575] LustreError: 53317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833357 with bad export cookie 13362208084197213738 [ 1844.004075] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1844.020082] LustreError: 53317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1844.284186] Lustre: server umount lustre-MDT0001 complete [ 1856.153072] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 1858.974715] Key type lgssc unregistered [ 1859.276183] LNet: 56867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.283373] LNetError: 56867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.326930] LNet: Removed LNI 192.168.204.118@tcp [ 1861.398331] Key type .llcrypt unregistered [ 1861.401078] Key type ._llcrypt unregistered [ 1886.574440] Key type ._llcrypt registered [ 1886.577314] Key type .llcrypt registered [ 1886.686261] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 1887.615275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1887.625301] alg: No test for adler32 (adler32-zlib) [ 1888.647689] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 1888.818951] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 1890.480181] Key type lgssc registered [ 1891.653170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1899.593373] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1899.628588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.232419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1920.352291] LustreError: 58249:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 1947.040256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.558602] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1964.037616] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1971.734666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.500427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.550688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1977.706273] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 1977.836414] LustreError: 58275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.863077] LustreError: 58275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1981.930632] LustreError: 58274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.958539] LustreError: 58274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1987.052658] LustreError: 59285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.083086] LustreError: 59285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1987.421163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.991336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.731154] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2001.785395] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2008.339560] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2013.670676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2013.673585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.681694] LustreError: Skipped 1 previous similar message [ 2013.684873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.719684] Lustre: Skipped 1 previous similar message [ 2016.945260] Lustre: server umount lustre-OST0000 complete [ 2039.644586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.655947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.663119] Lustre: Skipped 1 previous similar message [ 2044.385107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2044.395645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.415108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2046.065718] Lustre: server umount lustre-MDT0000 complete [ 2050.006583] LustreError: 58255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833563 with bad export cookie 14976100445747117393 [ 2050.012910] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.020377] LustreError: 58255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2064.353732] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2064.734584] Lustre: server umount lustre-MDT0001 complete [ 2076.066884] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2079.107895] Key type lgssc unregistered [ 2079.436890] LNet: 61995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.447405] LNetError: 61995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.465405] LNet: Removed LNI 192.168.204.118@tcp [ 2080.398817] Key type .llcrypt unregistered [ 2080.400931] Key type ._llcrypt unregistered [ 2103.435664] Key type ._llcrypt registered [ 2103.442695] Key type .llcrypt registered [ 2103.567531] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 2104.486983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.518093] alg: No test for adler32 (adler32-zlib) [ 2105.611939] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2105.936316] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 2107.634504] Key type lgssc registered [ 2108.646357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.473334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2115.492741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.980193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.701330] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2126.310889] LustreError: 63404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.322938] LustreError: 63404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2130.581664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.845590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.402890] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2143.658641] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2149.585537] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2156.488753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2156.722170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.849325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2164.154964] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2172.359479] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2190.817241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.823040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2190.838532] Lustre: Skipped 1 previous similar message [ 2190.857256] Lustre: Skipped 1 previous similar message [ 2195.580859] Lustre: server umount lustre-OST0000 complete [ 2201.066736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.072539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.082250] Lustre: Skipped 1 previous similar message [ 2201.090764] Lustre: Skipped 1 previous similar message [ 2206.030709] Lustre: server umount lustre-MDT0000 complete [ 2206.187695] LustreError: 64133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.214653] LustreError: 64133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2210.109364] LustreError: 63384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833724 with bad export cookie 12554352163559344813 [ 2210.142751] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.443230] Lustre: server umount lustre-MDT0001 complete [ 2221.423660] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2224.813238] Key type lgssc unregistered [ 2225.196546] LNet: 66959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.205024] LNetError: 66959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.232591] LNet: Removed LNI 192.168.204.118@tcp [ 2226.203186] Key type .llcrypt unregistered [ 2226.208266] Key type ._llcrypt unregistered [ 2253.597791] Key type ._llcrypt registered [ 2253.600444] Key type .llcrypt registered [ 2253.693538] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 2254.379451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2254.404861] alg: No test for adler32 (adler32-zlib) [ 2255.541740] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2255.847207] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 2257.488185] Key type lgssc registered [ 2258.718684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2267.997877] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 22:16:19 (1788833779) [ 2275.352937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.379277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.842260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.544447] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2287.075261] LustreError: 68659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.109477] LustreError: 68659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2291.759699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.189970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2297.397991] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2305.482388] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2310.766876] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2318.078173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.404641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.483482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2323.779462] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2330.490580] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2340.149209] Lustre: Failing over lustre-MDT0000 [ 2340.387797] Lustre: server umount lustre-MDT0000 complete [ 2342.885247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.887165] LustreError: 68655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.897147] Lustre: Skipped 2 previous similar messages [ 2342.909595] LustreError: 68655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2348.007808] LustreError: 68659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.022976] LustreError: 68659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2349.056713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.167705] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.286089] Lustre: *** cfs_fail_loc=135, val=0*** [ 2349.367728] LustreError: 71744:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2349.373975] LustreError: 71744:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.118@tcp: cfg command failed: rc = -2 [ 2349.379481] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2349.379481] [ 2349.386955] LustreError: MGC192.168.204.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2349.404817] LustreError: 71733:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2349.416236] LustreError: 71733:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2349.429201] LustreError: 71733:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2349.435995] LustreError: 71733:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2349.457677] Lustre: server umount lustre-MDT0000 complete [ 2349.460370] LustreError: 71733:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2351.013524] LustreError: 69633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.026720] LustreError: 69633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2356.101237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.194063] LustreError: 68658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.229902] LustreError: 68658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2356.339189] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2356.692141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.755695] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2358.722885] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2361.845354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2361.873734] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2361.907033] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2361.912976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2372.070591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.081289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.090840] Lustre: Skipped 1 previous similar message [ 2377.189257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.200082] Lustre: Skipped 2 previous similar messages [ 2382.310661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.316692] Lustre: Skipped 1 previous similar message [ 2384.352340] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2384.541112] Lustre: server umount lustre-OST0000 complete [ 2392.548619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.549618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.568776] Lustre: Skipped 1 previous similar message [ 2392.582098] Lustre: Skipped 1 previous similar message [ 2394.621183] Lustre: server umount lustre-MDT0000 complete [ 2397.665603] LustreError: 68653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.683554] LustreError: 68653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2399.021176] LustreError: 68641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833912 with bad export cookie 3039104957344841587 [ 2399.027846] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.030615] LustreError: 68641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2399.360198] Lustre: server umount lustre-MDT0001 complete [ 2411.032404] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2414.485686] Key type lgssc unregistered [ 2414.906182] LNet: 73525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.914459] LNetError: 73525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.928537] LNet: Removed LNI 192.168.204.118@tcp [ 2415.953715] Key type .llcrypt unregistered [ 2415.957723] Key type ._llcrypt unregistered [ 2444.869429] Key type ._llcrypt registered [ 2444.873449] Key type .llcrypt registered [ 2445.087957] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 2446.337798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.421870] alg: No test for adler32 (adler32-zlib) [ 2447.551245] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2447.843762] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 2449.496185] Key type lgssc registered [ 2450.887588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.059383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.098747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.615623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.317265] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2471.906957] LustreError: 74935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2471.932750] LustreError: 74935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2475.549554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.901634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.250882] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2489.353688] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2495.434857] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2502.452413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.820220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.937773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2508.813868] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2517.559591] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2526.611677] Lustre: Failing over lustre-MDT0000 [ 2526.951804] Lustre: server umount lustre-MDT0000 complete [ 2527.201265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2527.210177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2527.231336] LustreError: 75903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2529.760481] LustreError: 74931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.788324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.828175] Lustre: Skipped 1 previous similar message [ 2534.852989] LustreError: 76138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.880783] LustreError: 76138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2537.831130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.934189] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.115337] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2538.278248] LustreError: 78023:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2538.286140] LustreError: 78023:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.118@tcp: cfg command failed: rc = -9 [ 2538.308838] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2538.308838] [ 2538.320246] LustreError: MGC192.168.204.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2538.338529] LustreError: 78011:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2538.349268] LustreError: 78011:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2538.360632] LustreError: 78011:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2538.385636] LustreError: 78011:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2538.439470] Lustre: server umount lustre-MDT0000 complete [ 2538.445803] LustreError: 78011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2539.994341] LustreError: 74929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.025939] LustreError: 74929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2546.702109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.893064] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.905162] LustreError: 74926:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2547.265313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.317110] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2550.228300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2552.303447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2552.343621] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2552.409841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2552.962996] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2562.531757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.543111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.547894] Lustre: Skipped 1 previous similar message [ 2562.561081] Lustre: Skipped 1 previous similar message [ 2567.657155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.669474] Lustre: Skipped 1 previous similar message [ 2567.889299] Lustre: server umount lustre-OST0000 complete [ 2572.769317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.776133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.786206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.812345] Lustre: Skipped 1 previous similar message [ 2577.893894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.902619] Lustre: Skipped 2 previous similar messages [ 2578.510910] Lustre: server umount lustre-MDT0000 complete [ 2582.015626] LustreError: 74915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834095 with bad export cookie 634796943231501860 [ 2582.018784] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.026214] LustreError: 74915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.344584] Lustre: server umount lustre-MDT0001 complete [ 2593.455950] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2596.807721] Key type lgssc unregistered [ 2597.117787] LNet: 79804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.126504] LNetError: 79804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2597.156267] LNet: Removed LNI 192.168.204.118@tcp [ 2598.079488] Key type .llcrypt unregistered [ 2598.086080] Key type ._llcrypt unregistered [ 2610.373594] Key type ._llcrypt registered [ 2610.376534] Key type .llcrypt registered [ 2610.474521] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2610.730411] Key type .llcrypt unregistered [ 2610.732955] Key type ._llcrypt unregistered [ 2641.574903] Key type ._llcrypt registered [ 2641.580510] Key type .llcrypt registered [ 2641.696437] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 2643.044505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2643.056797] alg: No test for adler32 (adler32-zlib) [ 2644.194883] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2644.463686] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 2646.176227] Key type lgssc registered [ 2647.236260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2655.941652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2655.966546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.470672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2662.181290] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2668.002269] LustreError: 81702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.048879] LustreError: 81702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2671.643414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.009900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.934143] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2684.683984] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2690.933416] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2697.172328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2697.508139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.655607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2703.645452] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2711.590935] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2727.406542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.413299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.420285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.421144] Lustre: Skipped 1 previous similar message [ 2727.451714] Lustre: Skipped 1 previous similar message [ 2731.034335] Lustre: server umount lustre-OST0000 complete [ 2737.640945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.653362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.658490] Lustre: Skipped 1 previous similar message [ 2737.671038] Lustre: Skipped 1 previous similar message [ 2740.804269] Lustre: server umount lustre-MDT0000 complete [ 2742.753865] LustreError: 81702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.791201] LustreError: 81702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2744.235566] LustreError: 81682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834258 with bad export cookie 5650556625255759377 [ 2744.244938] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.692360] Lustre: server umount lustre-MDT0001 complete [ 2754.413943] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2757.088890] Key type lgssc unregistered [ 2757.427810] LNet: 85253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.435317] LNetError: 85253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.457767] LNet: Removed LNI 192.168.204.118@tcp [ 2758.387840] Key type .llcrypt unregistered [ 2758.390278] Key type ._llcrypt unregistered [ 2780.586280] Key type ._llcrypt registered [ 2780.591378] Key type .llcrypt registered [ 2780.715675] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 2781.524757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2781.628438] alg: No test for adler32 (adler32-zlib) [ 2782.704968] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2782.919651] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 2784.569536] Key type lgssc registered [ 2785.784597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2793.449673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2793.463873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.892563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.882874] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2804.196635] LustreError: 86663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.236044] LustreError: 86663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2806.128028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.408234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.681459] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2818.360529] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2824.167995] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2831.563776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.965747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.688612] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2840.185353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2846.322116] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2856.932452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 2856.933606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2856.936459] LustreError: Skipped 1 previous similar message [ 2856.958274] Lustre: Skipped 1 previous similar message [ 2856.964099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.977313] Lustre: Skipped 1 previous similar message [ 2862.055159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.077071] Lustre: Skipped 1 previous similar message [ 2867.171563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.176973] Lustre: Skipped 1 previous similar message [ 2871.265043] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2871.451180] Lustre: server umount lustre-OST0000 complete [ 2877.426967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.435313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.438315] Lustre: Skipped 1 previous similar message [ 2877.466949] Lustre: Skipped 1 previous similar message [ 2882.186495] Lustre: server umount lustre-MDT0000 complete [ 2882.530577] LustreError: 86663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.553789] LustreError: 86663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2886.334449] LustreError: 86643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834400 with bad export cookie 1995607229160251618 [ 2886.348118] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.349317] LustreError: 86643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2886.390621] LustreError: 87392:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2886.407842] LustreError: 87392:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2886.841348] Lustre: server umount lustre-MDT0001 complete [ 2896.591357] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 2899.570530] Key type lgssc unregistered [ 2899.839555] LNet: 90195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2899.846977] LNetError: 90195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.861653] LNet: Removed LNI 192.168.204.118@tcp [ 2900.634653] Key type .llcrypt unregistered [ 2900.637059] Key type ._llcrypt unregistered [ 2921.362360] Key type ._llcrypt registered [ 2921.367483] Key type .llcrypt registered [ 2921.487845] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 2922.083572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2922.106362] alg: No test for adler32 (adler32-zlib) [ 2923.117912] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 2923.334767] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 2924.968203] Key type lgssc registered [ 2925.863189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2933.240142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2933.270990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.668366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.164959] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2944.997810] LustreError: 91583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.016115] LustreError: 91583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2948.107250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.550475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.153943] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2960.630215] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2966.512422] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2975.195990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.575963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.651263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2983.379704] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 2991.220951] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3003.887566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3003.894657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.908591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.389660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.402659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.412451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.416165] Lustre: Skipped 1 previous similar message [ 3009.268360] Lustre: server umount lustre-OST0000 complete [ 3014.626943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.645680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.517766] Lustre: server umount lustre-MDT0000 complete [ 3019.748060] LustreError: 94162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.763493] LustreError: 94162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3023.063976] LustreError: 91563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834536 with bad export cookie 2900151635525852597 [ 3023.074277] LustreError: 91563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3023.075414] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.402285] Lustre: server umount lustre-MDT0001 complete [ 3034.410521] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 3038.291394] Key type lgssc unregistered [ 3038.669884] LNet: 95118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3038.678263] LNetError: 95118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3038.711408] LNet: Removed LNI 192.168.204.118@tcp [ 3039.864251] Key type .llcrypt unregistered [ 3039.867355] Key type ._llcrypt unregistered [ 3064.795052] Key type ._llcrypt registered [ 3064.796804] Key type .llcrypt registered [ 3065.079721] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 3066.474165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3066.646167] alg: No test for adler32 (adler32-zlib) [ 3067.691230] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 3067.932382] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 3069.560264] Key type lgssc registered [ 3070.859511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3079.151130] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3079.178488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.688466] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3099.809542] LustreError: 96501:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 3126.496306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.482681] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3141.827129] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3150.365662] Lustre: server umount lustre-OST0000 complete [ 3162.461909] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 22:31:13 (1788834673) [ 3170.534144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.023407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.592698] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3176.420536] LustreError: 98145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.446548] LustreError: 98145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3180.514723] LustreError: 98146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3184.772464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.166693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3190.108900] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3198.945068] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3205.694399] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3212.716564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.229753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.285546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3220.176173] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3229.065587] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3240.931476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3240.943403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.953276] LustreError: Skipped 1 previous similar message [ 3240.967449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.977232] Lustre: Skipped 1 previous similar message [ 3246.049852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3246.064818] Lustre: Skipped 1 previous similar message [ 3246.794538] Lustre: server umount lustre-OST0000 complete [ 3251.176464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.188192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3251.198656] Lustre: Skipped 2 previous similar messages [ 3256.290028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.299033] Lustre: Skipped 2 previous similar messages [ 3257.013985] Lustre: server umount lustre-MDT0000 complete [ 3260.023837] LustreError: 98125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834773 with bad export cookie 16709099394218687003 [ 3260.028060] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.033769] LustreError: 98125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3260.340200] Lustre: server umount lustre-MDT0001 complete [ 3271.353274] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 3273.969813] Key type lgssc unregistered [ 3274.385467] LNet: 101838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.395369] LNetError: 101838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.444862] LNet: Removed LNI 192.168.204.118@tcp [ 3276.669105] Key type .llcrypt unregistered [ 3276.672696] Key type ._llcrypt unregistered [ 3318.020947] Key type ._llcrypt registered [ 3318.025196] Key type .llcrypt registered [ 3318.147334] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 3318.901340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3319.023691] alg: No test for adler32 (adler32-zlib) [ 3320.291965] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 3320.571239] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 3322.376955] Key type lgssc registered [ 3323.685285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3333.860135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3333.887870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.751528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.388051] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3345.897304] LustreError: 104021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.932156] LustreError: 104021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3350.347661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.633147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3355.676412] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3364.714430] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3371.429886] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3378.708804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3379.121850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.163328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3386.094412] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3395.148986] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3405.793913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.800672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.808848] LustreError: Skipped 1 previous similar message [ 3405.811536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3405.819985] Lustre: Skipped 1 previous similar message [ 3405.842694] Lustre: Skipped 1 previous similar message [ 3410.919394] Lustre: server umount lustre-OST0000 complete [ 3416.034414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.039824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.046032] Lustre: Skipped 1 previous similar message [ 3416.061846] Lustre: Skipped 1 previous similar message [ 3420.662761] Lustre: server umount lustre-MDT0000 complete [ 3421.156773] LustreError: 105318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.175778] LustreError: 105318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3424.350227] LustreError: 104001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834938 with bad export cookie 2577670878736073285 [ 3424.362587] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.652451] Lustre: server umount lustre-MDT0001 complete [ 3434.287374] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 3436.933717] Key type lgssc unregistered [ 3437.217337] LNet: 107553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3437.227209] LNetError: 107553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3437.243568] LNet: Removed LNI 192.168.204.118@tcp [ 3438.298186] Key type .llcrypt unregistered [ 3438.301647] Key type ._llcrypt unregistered [ 3453.644748] Key type ._llcrypt registered [ 3453.653684] Key type .llcrypt registered [ 3453.802445] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 3454.670459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3454.692527] alg: No test for adler32 (adler32-zlib) [ 3455.666304] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 3455.943602] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 3457.592266] Key type lgssc registered [ 3458.505609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3465.454434] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3465.476174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.992470] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3486.114451] LustreError: 108787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 3512.801769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.709558] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3526.913648] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3533.685920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.961663] LustreError: MGC192.168.204.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3533.977470] LustreError: 109813:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3533.990330] LustreError: 109813:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3533.998454] LustreError: 109813:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3534.074409] Lustre: server umount lustre-MDT0000 complete [ 3534.084393] LustreError: 109813:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3554.272148] Lustre: 108123:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788835052/real 1788835052] req@00000000e942974c x1875729416457472/t0(0) o400->MGC192.168.204.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788835068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3554.295768] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.311704] LustreError: 108799:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3554.432931] Lustre: server umount lustre-OST0000 complete [ 3564.159194] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_hostid [ 3573.916130] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 3607.900286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.092861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3608.112844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3608.172812] Lustre: lustre-MDT0000: new disk, initializing [ 3608.253220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.271708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3612.737828] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3622.987619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.115526] Lustre: 112613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.168983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3623.172877] Lustre: Skipped 1 previous similar message [ 3623.232452] Lustre: lustre-MDT0001: new disk, initializing [ 3623.316283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3623.353636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3623.365500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3627.866950] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3637.793576] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3644.643784] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3652.406594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3652.710435] Lustre: lustre-OST0000: new disk, initializing [ 3652.716790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3652.724974] Lustre: 114580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3652.823331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.317845] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3659.889119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3659.910281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3660.004475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3668.526781] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3673.435642] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3673.655917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3679.528734] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3679.917871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3685.865804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3685.878135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.892465] Lustre: Skipped 1 previous similar message [ 3685.893789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3690.989259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3690.994408] Lustre: Skipped 2 previous similar messages [ 3696.101096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.112915] Lustre: Skipped 1 previous similar message [ 3697.120225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3697.255829] Lustre: server umount lustre-OST0000 complete [ 3701.219467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.233757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.339169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.346331] Lustre: Skipped 2 previous similar messages [ 3707.164528] Lustre: server umount lustre-MDT0000 complete [ 3710.978437] LustreError: 112605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835224 with bad export cookie 4009180394890612285 [ 3710.981163] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3710.989673] LustreError: 112605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.283440] Lustre: server umount lustre-MDT0001 complete [ 3719.966437] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 22:40:31 (1788835231) [ 3723.251817] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3737.782843] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_hostid [ 3747.517962] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 3788.174948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.473971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3788.523657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3788.612944] Lustre: lustre-MDT0000: new disk, initializing [ 3788.781895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.822269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3793.704582] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3804.300321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3804.439674] Lustre: 119478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3804.499685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3804.504255] Lustre: Skipped 1 previous similar message [ 3804.599867] Lustre: lustre-MDT0001: new disk, initializing [ 3804.684873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3804.735283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3804.759773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3808.689894] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3818.058786] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3823.582831] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3829.541602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.789595] Lustre: lustre-OST0000: new disk, initializing [ 3829.793618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3829.799663] Lustre: 121449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3829.889343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.618865] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3837.506766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3837.511968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3837.629647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3845.349619] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3850.480847] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3850.669319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3855.193915] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3855.355114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3858.401039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3858.407329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.415264] LustreError: Skipped 1 previous similar message [ 3858.420684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3858.433391] Lustre: Skipped 2 previous similar messages [ 3858.456542] Lustre: Skipped 1 previous similar message [ 3863.477918] Lustre: server umount lustre-OST0000 complete [ 3868.650968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.667910] Lustre: Skipped 1 previous similar message [ 3874.470309] Lustre: server umount lustre-MDT0000 complete [ 3879.363488] LustreError: 119471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835393 with bad export cookie 4009180394890613167 [ 3879.366723] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.375208] LustreError: 119471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3879.799858] Lustre: server umount lustre-MDT0001 complete [ 3889.628409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.052397] LustreError: 123668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.073507] LustreError: 123668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3890.118789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.016936] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3895.266332] LustreError: 123669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.395689] LustreError: 123668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.925630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.055299] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3918.066060] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3925.365201] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3933.926604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3940.609406] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 3949.504255] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3956.615551] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3960.804667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.805589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.811685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3960.818582] Lustre: Skipped 1 previous similar message [ 3960.829596] Lustre: Skipped 5 previous similar messages [ 3962.399732] Lustre: server umount lustre-OST0000 complete [ 3966.952242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3966.964352] LustreError: Skipped 1 previous similar message [ 3971.044400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.056451] Lustre: Skipped 1 previous similar message [ 3972.591436] Lustre: server umount lustre-MDT0000 complete [ 3976.164868] LustreError: 123668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.185001] LustreError: 123668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3976.572793] LustreError: 123650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835490 with bad export cookie 4009180394890614049 [ 3976.580376] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3976.812563] Lustre: server umount lustre-MDT0001 complete [ 3987.367599] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 3990.157970] Key type lgssc unregistered [ 3990.440612] LNet: 127297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3990.448843] LNetError: 127297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3990.469476] LNet: Removed LNI 192.168.204.118@tcp [ 3991.276521] Key type .llcrypt unregistered [ 3991.278602] Key type ._llcrypt unregistered [ 4007.975245] Key type ._llcrypt registered [ 4007.978056] Key type .llcrypt registered [ 4008.093427] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_hostid [ 4021.787134] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 4022.962736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.988734] alg: No test for adler32 (adler32-zlib) [ 4024.063179] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 4024.316230] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 4025.992832] Key type lgssc registered [ 4027.327280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4065.922828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4065.951623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.292629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4067.325497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4067.419814] Lustre: lustre-MDT0000: new disk, initializing [ 4067.513640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.528325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4071.761537] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4082.945252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.037517] Lustre: 130441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4083.093337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4083.095557] Lustre: Skipped 1 previous similar message [ 4083.192654] Lustre: lustre-MDT0001: new disk, initializing [ 4083.293996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4083.327580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4083.342837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4087.562653] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4096.257764] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4101.705578] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4108.000186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4108.303976] Lustre: lustre-OST0000: new disk, initializing [ 4108.306777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4108.311789] Lustre: 132412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4108.421912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.547255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4115.571155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4115.690211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4117.103333] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4130.312243] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4134.896710] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4135.115925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4139.176623] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4139.393492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4141.540548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.542608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.544126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.551712] Lustre: Skipped 1 previous similar message [ 4141.563457] Lustre: Skipped 1 previous similar message [ 4146.663227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4146.672174] Lustre: Skipped 1 previous similar message [ 4147.613339] Lustre: server umount lustre-OST0000 complete [ 4151.785335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.817512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.825189] Lustre: Skipped 1 previous similar message [ 4156.900367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4156.912628] Lustre: Skipped 1 previous similar message [ 4157.090483] Lustre: server umount lustre-MDT0000 complete [ 4161.076294] LustreError: 131239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835675 with bad export cookie 9615162410977366131 [ 4161.082759] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4161.085224] LustreError: 131239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4161.444209] Lustre: server umount lustre-MDT0001 complete [ 4169.220039] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:48:00 (1788835680) [ 4175.784034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.444254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.081730] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4181.477575] LustreError: 134827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.497735] LustreError: 134827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4186.594139] LustreError: 134826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4190.690456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.226236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.391530] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4205.273819] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4211.645942] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4213.656351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.660751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4213.668528] Lustre: Skipped 2 previous similar messages [ 4213.679114] Lustre: Skipped 1 previous similar message [ 4219.926398] Lustre: server umount lustre-MDT0000 complete [ 4221.922828] LustreError: 134823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.942380] LustreError: 134823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4223.776300] LustreError: 134807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835737 with bad export cookie 9615162410977367013 [ 4223.788499] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.794994] LustreError: 134807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4224.196550] Lustre: server umount lustre-MDT0001 complete [ 4233.318481] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:49:04 (1788835744) [ 4240.023368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4255.585506] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4260.705496] LustreError: 137242:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 4286.432758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.365926] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4300.725350] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4303.846651] Lustre: server umount lustre-OST0000 complete [ 4314.457310] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:50:26 (1788835826) [ 4322.429528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.690211] LustreError: 138688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.749928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.416852] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4327.905333] LustreError: 138689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.907273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4340.699987] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4349.312524] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4356.008273] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4363.644741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4363.995223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.003206] Lustre: Skipped 1 previous similar message [ 4370.681502] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4379.079536] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4391.922444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.958685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4391.963899] Lustre: Skipped 3 previous similar messages [ 4392.209777] Lustre: server umount lustre-MDT0000 complete [ 4392.939310] LustreError: 138684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.973567] LustreError: 138684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4395.933491] LustreError: 138669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835909 with bad export cookie 9615162410977367678 [ 4395.954739] LustreError: 138669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4395.962469] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4396.307316] Lustre: server umount lustre-MDT0001 complete [ 4409.434668] Lustre: server umount lustre-OST0000 complete [ 4417.995906] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:52:09 (1788835929) [ 4424.578698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.952323] LustreError: 142359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4430.179149] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4438.796747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.053735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.058464] Lustre: Skipped 1 previous similar message [ 4444.191947] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4450.462983] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4455.279874] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4461.052268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.958469] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4469.623869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4474.446497] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4478.840871] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4479.132775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4483.033541] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4483.282987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4489.186813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4489.197577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.214939] Lustre: Skipped 2 previous similar messages [ 4489.219864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4490.210912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4491.508562] Lustre: server umount lustre-OST0000 complete [ 4495.330478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4501.283450] Lustre: server umount lustre-MDT0000 complete [ 4504.649115] LustreError: 144205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836018 with bad export cookie 9615162410977368749 [ 4504.649937] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4504.662718] LustreError: 144205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4504.952676] Lustre: server umount lustre-MDT0001 complete [ 4512.257155] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:53:44 (1788836024) [ 4518.695357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4534.241643] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4539.360651] LustreError: 146520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 4573.390559] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4582.270881] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4590.268872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.820850] LustreError: 146545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.848764] LustreError: 146545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4590.929810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.934919] Lustre: Skipped 2 previous similar messages [ 4591.044323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4595.579663] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4605.059681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.259708] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4621.610082] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4629.458533] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4635.469311] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4635.780980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4640.635590] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4640.867345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4646.371974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4646.381352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.390323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.399703] Lustre: Skipped 5 previous similar messages [ 4646.419543] Lustre: Skipped 8 previous similar messages [ 4649.342056] Lustre: server umount lustre-MDT0000 complete [ 4653.370450] LustreError: 146527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836167 with bad export cookie 9615162410977369771 [ 4653.379062] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.379627] LustreError: 146527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4653.754392] Lustre: server umount lustre-MDT0001 complete [ 4667.971926] Lustre: server umount lustre-OST0000 complete [ 4679.445695] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:56:30 (1788836190) [ 4687.822764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4703.521638] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4708.642499] LustreError: 150717:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 4740.077248] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4747.738487] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4755.481648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.708368] LustreError: 128666:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ccaa1595 x1875730013250688/t0(0) o250->MGC192.168.204.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4756.297384] LustreError: 150742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.327332] LustreError: 150742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4756.613669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4762.743145] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4771.345704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.306581] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4784.360967] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4791.263786] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4798.981241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4799.157242] Lustre: lustre-OST0001: new disk, initializing [ 4799.160257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4799.168223] Lustre: 153594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4805.124762] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4805.736241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4805.748406] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4805.857980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4814.838949] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.094253] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4820.369568] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4825.852933] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4826.162930] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4831.723505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.736742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.741385] Lustre: Skipped 1 previous similar message [ 4831.752811] Lustre: Skipped 1 previous similar message [ 4834.290681] Lustre: server umount lustre-OST0000 complete [ 4841.976031] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4844.769365] Lustre: server umount lustre-OST0001 complete [ 4855.631678] Lustre: server umount lustre-MDT0000 complete [ 4859.874662] LustreError: 150724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836373 with bad export cookie 9615162410977370723 [ 4859.879749] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4859.891866] LustreError: 150724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4860.197127] Lustre: server umount lustre-MDT0001 complete [ 4883.548691] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:59:54 (1788836394) [ 4885.423142] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4887.188090] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4889.082686] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:00:00 (1788836400) [ 4895.870120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.187920] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4896.217905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4896.419985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.425445] Lustre: Skipped 5 previous similar messages [ 4901.454956] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4911.109770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.259715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4911.283462] Lustre: 157153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4916.758932] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4924.913483] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4931.584764] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4939.260539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4939.538622] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4948.748106] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 4948.767787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4958.084783] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4964.011328] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4964.295720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4969.605149] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4969.976111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4974.565332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.569041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4974.588230] Lustre: Skipped 4 previous similar messages [ 4974.597036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.599273] LustreError: Skipped 1 previous similar message [ 4974.621296] Lustre: Skipped 5 previous similar messages [ 4986.337154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4986.565756] Lustre: server umount lustre-OST0000 complete [ 5010.278375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.724175] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.736460] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5014.739407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.400934] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5025.905712] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5031.307502] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5031.537595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5036.630618] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5036.914024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5043.394980] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9edc85318000.ost_server_uuid 50 [ 5045.117842] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9edc85318000.ost_server_uuid in FULL state after 0 sec [ 5045.305599] LustreError: 159202:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.18@tcp) returned error from blocking AST (req@00000000143085c0 x1875730013361408 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000cee41e6f/0x856feb44fbfeabbd lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.18@tcp remote: 0xa54d1035f9b1ba13 expref: 5 pid: 157161 timeout: 5145 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5045.348520] LustreError: lustre-MDT0000: A client on nid 192.168.204.18@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5045.360689] LustreError: 157149:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.18@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000cee41e6f/0x856feb44fbfeabbd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.18@tcp remote: 0xa54d1035f9b1ba13 expref: 6 pid: 157161 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5050.849165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5050.853566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.865512] LustreError: Skipped 1 previous similar message [ 5050.885099] Lustre: Skipped 2 previous similar messages [ 5055.871629] Lustre: server umount lustre-OST0000 complete [ 5074.400102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5074.793666] Lustre: server umount lustre-MDT0000 complete [ 5075.435667] LustreError: 157161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5075.457132] LustreError: 157161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5079.312417] LustreError: 159017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836593 with bad export cookie 9615162410977371906 [ 5079.322449] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5079.342644] LustreError: 159017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5085.860355] Lustre: server umount lustre-MDT0001 complete [ 5098.311053] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5100.924102] Key type lgssc unregistered [ 5101.224748] LNet: 163295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5101.235339] LNetError: 163295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5101.275410] LNet: Removed LNI 192.168.204.118@tcp [ 5102.112342] Key type .llcrypt unregistered [ 5102.114759] Key type ._llcrypt unregistered [ 5131.469297] Key type ._llcrypt registered [ 5131.473356] Key type .llcrypt registered [ 5131.601057] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 5132.851033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5133.001589] alg: No test for adler32 (adler32-zlib) [ 5134.325986] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 5134.651807] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 5136.365923] Key type lgssc registered [ 5137.616979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5145.844501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5145.871590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.469537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.264675] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5157.864259] LustreError: 164704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.884391] LustreError: 164704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5160.850716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.312939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5165.839825] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5174.230553] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5179.339774] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5186.373908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5186.651303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.830191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5193.226829] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5201.029152] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5206.604450] Lustre: Failing over lustre-MDT0000 [ 5206.896240] Lustre: server umount lustre-MDT0000 complete [ 5207.523971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5207.530572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.549636] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.547111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.554546] LustreError: 164704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.611111] LustreError: 164704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5213.671890] LustreError: 165435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5213.715467] LustreError: 165435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5218.786527] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.813246] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5219.270182] LustreError: 164684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.18@tcp arrived at 1788836733 with bad export cookie 16840110153599988042 [ 5224.928388] Lustre: 164013:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788836722/real 1788836722] req@000000008746abed x1875731177039232/t0(0) o400->MGC192.168.204.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788836738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5224.978293] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5230.053087] LustreError: 165718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.091261] LustreError: 165718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5231.032816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.152139] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe9b416f643c6b1ac [ 5234.165590] Lustre: MGC192.168.204.118@tcp: Connection restored to 0@lo (at 0@lo) [ 5234.384575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.501902] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5239.812397] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5239.838859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5240.045112] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5242.402677] Lustre: lustre-MDT0000: Denying connection for new client 1c4d8f23-563d-4919-9259-e4d5a82c8ec8 (at 192.168.204.18@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5254.626164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5254.630480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5254.642570] Lustre: Skipped 1 previous similar message [ 5254.650139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5256.883593] Lustre: server umount lustre-OST0000 complete [ 5260.944195] LustreError: 168438:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5260.956819] Lustre: 167784:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5260.963789] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5260.981085] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5260.994528] LustreError: 167784:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003a8cd7d3 x1875731177059328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5261.017548] LustreError: 167784:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5261.037973] LustreError: 167784:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5261.053549] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5261.369835] Lustre: server umount lustre-MDT0000 complete [ 5265.353756] LustreError: 164684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836779 with bad export cookie 16840110153599988140 [ 5265.356065] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5265.360100] LustreError: 164684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5265.372015] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.418089] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5267.796611] Lustre: server umount lustre-MDT0001 complete [ 5281.398786] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5284.054477] Key type lgssc unregistered [ 5284.367428] LNet: 169189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5284.376933] LNetError: 169189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5284.398942] LNet: Removed LNI 192.168.204.118@tcp [ 5285.234118] Key type .llcrypt unregistered [ 5285.236170] Key type ._llcrypt unregistered [ 5310.710656] Key type ._llcrypt registered [ 5310.712664] Key type .llcrypt registered [ 5310.849892] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 5311.647776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5311.685805] alg: No test for adler32 (adler32-zlib) [ 5312.720611] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 5313.014507] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 5314.704182] Key type lgssc registered [ 5315.818656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5323.686406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5323.705872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.176739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.403437] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5335.528030] LustreError: 170610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.553182] LustreError: 170610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5337.309655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.639476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.102815] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5350.345784] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5359.587433] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5371.219627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5371.712190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.500593] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5390.247227] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5395.334354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5403.621245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5403.630690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.657911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.665407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5405.668136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.669656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.678301] Lustre: Skipped 1 previous similar message [ 5408.110848] Lustre: server umount lustre-OST0000 complete [ 5414.377453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5414.393173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.411688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5418.945605] Lustre: server umount lustre-MDT0000 complete [ 5421.027241] LustreError: 170605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5421.063842] LustreError: 170605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5425.423576] LustreError: 170593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836939 with bad export cookie 14545652802992636408 [ 5425.462895] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.319946] Lustre: server umount lustre-MDT0001 complete [ 5441.989591] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5445.132650] Key type lgssc unregistered [ 5445.423056] LNet: 174144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5445.433709] LNetError: 174144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5445.445269] LNet: Removed LNI 192.168.204.118@tcp [ 5446.506629] Key type .llcrypt unregistered [ 5446.509707] Key type ._llcrypt unregistered [ 5486.050953] Key type ._llcrypt registered [ 5486.053204] Key type .llcrypt registered [ 5486.178229] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 5487.212139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5487.242456] alg: No test for adler32 (adler32-zlib) [ 5488.432557] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 5488.728749] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 5490.440203] Key type lgssc registered [ 5491.444723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5498.750146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5498.772247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.231877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.851336] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5510.631641] LustreError: 175841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.654420] LustreError: 175841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5514.217430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.741198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5520.468102] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5529.113910] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5535.076508] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5543.034252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5543.572361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.591949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5548.539795] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5555.997435] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5570.025928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5570.036351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5570.067211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5570.551994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.656245] LustreError: 178335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.854694] Lustre: server umount lustre-OST0000 complete [ 5580.772196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.777239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5580.794889] Lustre: Skipped 2 previous similar messages [ 5580.811041] Lustre: Skipped 2 previous similar messages [ 5585.892090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.900328] Lustre: Skipped 1 previous similar message [ 5586.238154] Lustre: server umount lustre-MDT0000 complete [ 5589.484892] LustreError: 175824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837103 with bad export cookie 95072060114489338 [ 5589.504824] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.906372] Lustre: server umount lustre-MDT0001 complete [ 5606.069645] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5609.854824] Key type lgssc unregistered [ 5610.374307] LNet: 179396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5610.383295] LNetError: 179396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5611.432113] LNet: Removed LNI 192.168.204.118@tcp [ 5612.567697] Key type .llcrypt unregistered [ 5612.570363] Key type ._llcrypt unregistered [ 5640.970110] Key type ._llcrypt registered [ 5640.974259] Key type .llcrypt registered [ 5641.210766] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 5642.476810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5642.622839] alg: No test for adler32 (adler32-zlib) [ 5643.697754] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 5644.015089] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 5645.805488] Key type lgssc registered [ 5647.652921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5664.588940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5664.617204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.054586] Lustre: *** cfs_fail_loc=135, val=0*** [ 5666.184290] LustreError: 180917:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5666.192184] LustreError: 180917:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.118@tcp: cfg command failed: rc = -2 [ 5666.205418] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5666.205418] [ 5666.213249] LustreError: MGC192.168.204.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5666.227572] LustreError: 180877:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5666.237841] LustreError: 180877:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5666.250024] LustreError: 180877:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5666.264197] LustreError: 180877:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5666.357997] Lustre: server umount lustre-MDT0000 complete [ 5666.360849] LustreError: 180877:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5673.950541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.448662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.437548] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5684.707732] LustreError: 181289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.736869] LustreError: 181289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5688.701664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.072242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5694.523374] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5702.996431] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5709.539652] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.830359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.833349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.841206] Lustre: Skipped 1 previous similar message [ 5719.011673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5719.024825] Lustre: Skipped 2 previous similar messages [ 5724.131895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5724.139692] Lustre: Skipped 1 previous similar message [ 5726.176325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5726.506983] Lustre: server umount lustre-MDT0000 complete [ 5729.250388] LustreError: 181283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.269334] LustreError: 181283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5730.287695] LustreError: 181271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837244 with bad export cookie 18037174462120362887 [ 5730.298556] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5730.307565] LustreError: 181271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5730.639875] Lustre: server umount lustre-MDT0001 complete [ 5743.072434] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5746.249753] Key type lgssc unregistered [ 5746.672739] LNet: 183613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5746.692778] LNetError: 183613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5746.724209] LNet: Removed LNI 192.168.204.118@tcp [ 5747.842824] Key type .llcrypt unregistered [ 5747.850736] Key type ._llcrypt unregistered [ 5772.845384] Key type ._llcrypt registered [ 5772.852243] Key type .llcrypt registered [ 5773.010324] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5773.279577] Key type .llcrypt unregistered [ 5773.283743] Key type ._llcrypt unregistered [ 5792.028488] Key type ._llcrypt registered [ 5792.034657] Key type .llcrypt registered [ 5792.142806] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 5793.250135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5793.420440] alg: No test for adler32 (adler32-zlib) [ 5794.508722] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 5794.733183] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 5796.425988] Key type lgssc registered [ 5797.865911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5805.576468] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5805.594598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5821.088424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5826.208802] LustreError: 185738:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.118@tcp: failed processing log, type 4: rc = -110 [ 5852.960577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.572781] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5869.522153] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5877.640307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.395350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.504590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5883.876838] LustreError: 185763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.891831] LustreError: 185763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5884.361786] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5887.984367] LustreError: 186778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.025287] LustreError: 186778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5893.106314] LustreError: 186777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5893.128257] LustreError: 186777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5894.826818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.451464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5900.052429] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 5911.582547] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5918.103428] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5921.216473] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5934.565119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5934.576575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.596652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.618639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.620695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.986846] Lustre: server umount lustre-OST0000 complete [ 5946.852175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.854851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5946.855770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5946.855776] Lustre: Skipped 1 previous similar message [ 5946.870602] Lustre: Skipped 1 previous similar message [ 5949.404450] Lustre: server umount lustre-MDT0000 complete [ 5951.970067] LustreError: 186776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.996043] LustreError: 186776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5952.979054] LustreError: 187519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837466 with bad export cookie 13562280041711852474 [ 5952.990907] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5952.998436] LustreError: 187519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5953.538216] Lustre: server umount lustre-MDT0001 complete [ 5967.132262] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 5970.139738] Key type lgssc unregistered [ 5970.466398] LNet: 189709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5970.471846] LNetError: 189709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5971.499570] LNet: Removed LNI 192.168.204.118@tcp [ 5972.308584] Key type .llcrypt unregistered [ 5972.310818] Key type ._llcrypt unregistered [ 5998.465442] Key type ._llcrypt registered [ 5998.467568] Key type .llcrypt registered [ 5998.570853] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 5999.954958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6000.046181] alg: No test for adler32 (adler32-zlib) [ 6001.161474] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 6001.415534] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 6003.112196] Key type lgssc registered [ 6004.248481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6013.015595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6013.032762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.462071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.699341] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6024.675360] LustreError: 191121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.688573] LustreError: 191121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6027.920688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6028.257950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.533720] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6041.928347] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6048.565396] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6056.752227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6057.138158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.196403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6065.019715] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6074.019331] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6080.465938] Lustre: Failing over lustre-MDT0000 [ 6080.884146] Lustre: server umount lustre-MDT0000 complete [ 6082.016898] LustreError: 191118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.076489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.084224] LustreError: 191849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.098527] Lustre: Skipped 1 previous similar message [ 6087.099751] LustreError: 191116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.127753] LustreError: 191116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6092.218282] LustreError: 191116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.241247] LustreError: 191116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6100.448155] Lustre: 190432:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837598/real 1788837598] req@000000009db77b85 x1875732086156288/t0(0) o400->MGC192.168.204.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788837614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6100.510573] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6100.531342] LustreError: 191118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.554384] LustreError: 191118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6100.854254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.699275] LustreError: 190429:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fc3a242c x1875732086161408/t0(0) o250->MGC192.168.204.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6111.181704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.270601] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6113.717180] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6115.820662] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6116.352886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6116.424435] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6116.497118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6121.753602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6127.951177] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6135.266208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6135.277939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.293185] Lustre: Skipped 1 previous similar message [ 6135.310782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.787891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6136.805649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.814661] Lustre: Skipped 2 previous similar messages [ 6137.398154] Lustre: server umount lustre-OST0000 complete [ 6141.924244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.926439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.931499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6141.948665] Lustre: Skipped 2 previous similar messages [ 6141.968529] Lustre: Skipped 1 previous similar message [ 6147.051661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6147.068791] Lustre: Skipped 1 previous similar message [ 6147.439409] Lustre: server umount lustre-MDT0000 complete [ 6150.699370] LustreError: 191102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837664 with bad export cookie 12374971638143541716 [ 6150.703758] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.709451] LustreError: 191102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6151.068678] Lustre: server umount lustre-MDT0001 complete [ 6162.960963] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 6165.862774] Key type lgssc unregistered [ 6166.137153] LNet: 195960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6166.157289] LNetError: 195960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6166.180342] LNet: Removed LNI 192.168.204.118@tcp [ 6167.238578] Key type .llcrypt unregistered [ 6167.241587] Key type ._llcrypt unregistered [ 6193.473461] Key type ._llcrypt registered [ 6193.475382] Key type .llcrypt registered [ 6193.597851] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 6195.187850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6195.282752] alg: No test for adler32 (adler32-zlib) [ 6196.454539] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 6196.833489] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 6198.584209] Key type lgssc registered [ 6200.183051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6209.729164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6209.770120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.267419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6217.432587] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6221.808770] LustreError: 197371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.851822] LustreError: 197371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6222.821626] LustreError: 197372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.939275] LustreError: 197371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.111774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.614962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6233.988655] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6243.005450] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6250.633445] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6259.140387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6259.593142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.813518] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6275.163283] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6282.111720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6284.030149] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6289.751804] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6307.366769] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6320.097451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6320.104409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6320.121894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6323.169389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6323.171699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6323.186447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.233392] Lustre: Skipped 1 previous similar message [ 6325.472747] Lustre: server umount lustre-OST0000 complete [ 6330.850305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6330.855808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.873844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.418684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.432204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6338.538170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6338.545187] Lustre: Skipped 2 previous similar messages [ 6344.160168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6344.423342] Lustre: server umount lustre-MDT0000 complete [ 6348.403612] LustreError: 197352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837862 with bad export cookie 782793395544001320 [ 6348.414449] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6348.821732] Lustre: server umount lustre-MDT0001 complete [ 6362.003487] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 6365.168355] Key type lgssc unregistered [ 6365.556925] LNet: 201245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6365.571076] LNetError: 201245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6366.636492] LNet: Removed LNI 192.168.204.118@tcp [ 6367.612204] Key type .llcrypt unregistered [ 6367.615123] Key type ._llcrypt unregistered [ 6389.363764] Key type ._llcrypt registered [ 6389.369204] Key type .llcrypt registered [ 6389.575696] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 6390.911177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6390.964163] alg: No test for adler32 (adler32-zlib) [ 6392.469983] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 6392.966478] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 6394.760200] Key type lgssc registered [ 6396.028245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6404.702465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6404.722997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.284107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.195459] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6416.865669] LustreError: 202506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.882700] LustreError: 202506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6420.188456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.622776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.311108] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6433.886765] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6440.916119] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6449.359522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.701269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6451.765261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6455.683763] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6464.981558] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6471.912810] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6481.514401] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6488.194447] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6493.651696] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6501.859765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6501.868198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6501.877149] Lustre: Skipped 1 previous similar message [ 6501.901312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6506.980924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6507.008460] Lustre: Skipped 2 previous similar messages [ 6512.111824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6512.121281] Lustre: Skipped 1 previous similar message [ 6515.168369] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6515.329338] Lustre: server umount lustre-OST0000 complete [ 6522.339563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.341354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6522.351820] Lustre: Skipped 1 previous similar message [ 6522.361784] Lustre: Skipped 1 previous similar message [ 6525.664240] Lustre: server umount lustre-MDT0000 complete [ 6527.458684] LustreError: 203234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6527.471353] LustreError: 203234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6529.907066] LustreError: 202489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838043 with bad export cookie 15072915869383433280 [ 6529.921735] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.927949] LustreError: 202489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6530.373267] Lustre: server umount lustre-MDT0001 complete [ 6543.461792] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 6545.984576] Key type lgssc unregistered [ 6546.337974] LNet: 207237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6546.350120] LNetError: 207237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6546.372613] LNet: Removed LNI 192.168.204.118@tcp [ 6547.447349] Key type .llcrypt unregistered [ 6547.454577] Key type ._llcrypt unregistered [ 6575.092647] Key type ._llcrypt registered [ 6575.095838] Key type .llcrypt registered [ 6575.184605] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 6575.938516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6576.013964] alg: No test for adler32 (adler32-zlib) [ 6577.083593] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 6577.412133] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 6579.112204] Key type lgssc registered [ 6580.268168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6588.009915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6588.030995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6589.692297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.558345] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6599.148768] LustreError: 208647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6599.164834] LustreError: 208647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6604.261989] LustreError: 208648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.792188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.392943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6610.854914] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6619.415508] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6625.705860] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6633.411158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6634.034648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.066487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6641.667946] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6650.830221] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.817556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6670.829738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.850661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.490557] Lustre: server umount lustre-OST0000 complete [ 6676.961544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6676.974061] LustreError: Skipped 1 previous similar message [ 6676.984994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.001569] Lustre: Skipped 1 previous similar message [ 6677.010172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.014308] Lustre: Skipped 1 previous similar message [ 6681.058190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.059339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.096272] Lustre: Skipped 1 previous similar message [ 6682.439589] LustreError: 211582:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f79b37a6) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6684.586214] Lustre: server umount lustre-MDT0000 complete [ 6686.180648] LustreError: 208647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.206123] LustreError: 208647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6688.699239] LustreError: 208628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838202 with bad export cookie 5229587266567491179 [ 6688.704254] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.713888] LustreError: 208628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6689.175612] Lustre: server umount lustre-MDT0001 complete [ 6703.434811] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 6706.940867] Key type lgssc unregistered [ 6707.217051] LNet: 212336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6707.241622] LNetError: 212336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6707.270263] LNet: Removed LNI 192.168.204.118@tcp [ 6708.267983] Key type .llcrypt unregistered [ 6708.270573] Key type ._llcrypt unregistered [ 6728.253988] Key type ._llcrypt registered [ 6728.256775] Key type .llcrypt registered [ 6728.371536] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 6729.473807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6729.517772] alg: No test for adler32 (adler32-zlib) [ 6730.646629] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 6730.940320] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 6732.648972] Key type lgssc registered [ 6733.895606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6742.067565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6742.096238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.807976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6748.712090] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6753.252520] LustreError: 213597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.267111] LustreError: 213597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6757.152534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.535153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6762.869920] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6771.698566] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.945989] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6784.181761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.532614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.677637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6791.651842] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6800.221874] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6814.179626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6814.188156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.211032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.273398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6818.289038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.299875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6818.317965] Lustre: Skipped 1 previous similar message [ 6819.879624] Lustre: server umount lustre-OST0000 complete [ 6828.514491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.517451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.535581] Lustre: Skipped 1 previous similar message [ 6830.340460] Lustre: server umount lustre-MDT0000 complete [ 6833.633638] LustreError: 214727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.648173] LustreError: 214727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6834.085568] LustreError: 213579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838348 with bad export cookie 17222614640021142073 [ 6834.087045] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6834.096094] LustreError: 213579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6834.352572] Lustre: server umount lustre-MDT0001 complete [ 6844.858582] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 6847.898656] Key type lgssc unregistered [ 6848.405879] LNet: 217281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6848.422869] LNetError: 217281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6848.460426] LNet: Removed LNI 192.168.204.118@tcp [ 6849.799240] Key type .llcrypt unregistered [ 6849.802547] Key type ._llcrypt unregistered [ 6872.600976] Key type ._llcrypt registered [ 6872.604275] Key type .llcrypt registered [ 6872.764479] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 6874.058475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6874.122339] alg: No test for adler32 (adler32-zlib) [ 6875.112974] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 6875.375551] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 6877.120301] Key type lgssc registered [ 6878.536691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6887.402872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6887.422578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.944751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.930664] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6898.150416] LustreError: 218544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.177148] LustreError: 218544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6903.276042] LustreError: 218545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.746067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.220491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6912.537261] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6920.956797] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6928.315573] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6935.215397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6935.701508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.653446] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6951.133466] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6958.201466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6970.041548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.120739] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6970.350902] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.819774] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 6983.357798] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6985.069547] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6997.212630] Lustre: Permanently deactivating lustre-OST0001 [ 6997.250497] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7005.287162] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7012.751405] Lustre: Permanently reactivating lustre-OST0001 [ 7012.766434] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7012.773889] Lustre: Skipped 2 previous similar messages [ 7013.508069] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.526380] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7013.535122] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7013.554321] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7013.581351] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7022.980493] Lustre: server umount lustre-OST0001 complete [ 7024.102250] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7024.119862] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.133604] Lustre: Skipped 1 previous similar message [ 7024.153300] LustreError: 220371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.173294] LustreError: 220371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7029.734802] LustreError: 221043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.742157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.758938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7029.780565] LustreError: 221043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7029.789738] Lustre: Skipped 2 previous similar messages [ 7033.721910] Lustre: server umount lustre-OST0000 complete [ 7039.974380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.007889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7040.016783] Lustre: Skipped 2 previous similar messages [ 7043.974644] Lustre: server umount lustre-MDT0000 complete [ 7045.091373] LustreError: 218544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7045.106302] LustreError: 218544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7048.049059] LustreError: 221409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838561 with bad export cookie 14209804771206612327 [ 7048.059144] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.066595] LustreError: 221409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7048.392655] Lustre: server umount lustre-MDT0001 complete [ 7060.588858] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 7063.390631] Key type lgssc unregistered [ 7063.683076] LNet: 223615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7063.691975] LNetError: 223615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7063.711332] LNet: Removed LNI 192.168.204.118@tcp [ 7064.555574] Key type .llcrypt unregistered [ 7064.557713] Key type ._llcrypt unregistered [ 7110.380606] Key type ._llcrypt registered [ 7110.385078] Key type .llcrypt registered [ 7110.593145] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 7112.113639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7112.182348] alg: No test for adler32 (adler32-zlib) [ 7113.330806] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 7113.512642] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 7115.170376] Key type lgssc registered [ 7116.437411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7125.167641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7125.211674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7126.556881] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7126.623255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7126.863483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.876155] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 7142.442231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.546888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7142.617978] Lustre: 225815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7143.051784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7148.017763] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 7157.082719] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7163.945670] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7171.113256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7171.335439] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7171.556318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7177.909541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7177.992819] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 7185.224692] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7188.103641] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7194.828970] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7201.417863] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7208.779131] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7220.500536] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7220.516892] Lustre: Skipped 1 previous similar message [ 7243.115878] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7243.127088] Lustre: Skipped 2 previous similar messages [ 7279.264029] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7279.275629] Lustre: Skipped 4 previous similar messages [ 7350.557231] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7350.562401] Lustre: Skipped 7 previous similar messages [ 7362.032214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7362.047491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.078378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7364.360891] Lustre: server umount lustre-OST0000 complete [ 7368.164558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7368.170376] LustreError: Skipped 1 previous similar message [ 7368.175617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.188563] Lustre: Skipped 1 previous similar message [ 7368.195882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7368.204109] Lustre: Skipped 1 previous similar message [ 7372.769950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.779797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.309296] Lustre: server umount lustre-MDT0000 complete [ 7378.248827] LustreError: 225810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838892 with bad export cookie 9651299536596997106 [ 7378.257973] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.268436] LustreError: 225810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7378.813846] Lustre: server umount lustre-MDT0001 complete [ 7391.654764] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 7394.449274] Key type lgssc unregistered [ 7394.718829] LNet: 230620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7394.727832] LNetError: 230620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7394.749401] LNet: Removed LNI 192.168.204.118@tcp [ 7395.585562] Key type .llcrypt unregistered [ 7395.588718] Key type ._llcrypt unregistered [ 7422.731532] Key type ._llcrypt registered [ 7422.734704] Key type .llcrypt registered [ 7422.914490] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 7424.000087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7424.112314] alg: No test for adler32 (adler32-zlib) [ 7425.224641] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 7425.584354] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 7427.348462] Key type lgssc registered [ 7428.751765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7436.763256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7436.786254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.314498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.321122] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 7448.549878] LustreError: 232031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.586528] LustreError: 232031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7452.430737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.839363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7457.451980] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 7465.363345] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7472.450382] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7479.921271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7480.289571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.388525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7487.414327] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_default_debug -1 all [ 7496.064950] Lustre: DEBUG MARKER: oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7518.187792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7518.191391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.228783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.079270] Lustre: server umount lustre-OST0000 complete [ 7528.165588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.178811] Lustre: Skipped 1 previous similar message [ 7528.183214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7528.187469] Lustre: Skipped 2 previous similar messages [ 7531.334343] Lustre: server umount lustre-MDT0000 complete [ 7533.543489] LustreError: 232031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.559827] LustreError: 232031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7535.545960] LustreError: 232013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839049 with bad export cookie 13120926360839860220 [ 7535.556992] LustreError: MGC192.168.204.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.573881] LustreError: 232013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7536.083838] Lustre: server umount lustre-MDT0001 complete [ 7548.315146] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 7551.367992] Key type lgssc unregistered [ 7551.699099] LNet: 235726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7551.707454] LNetError: 235726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7551.733519] LNet: Removed LNI 192.168.204.118@tcp [ 7552.869375] Key type .llcrypt unregistered [ 7552.878151] Key type ._llcrypt unregistered [ 7579.615259] Key type ._llcrypt registered [ 7579.617909] Key type .llcrypt registered [ 7579.742820] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 7581.117100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7581.166735] alg: No test for adler32 (adler32-zlib) [ 7582.293861] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 7582.605751] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 7584.305468] Key type lgssc registered [ 7585.966379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7858.726867] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 7861.900782] Key type lgssc unregistered [ 7862.367293] LNet: 237748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7862.374202] LNetError: 237748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7862.393359] LNet: Removed LNI 192.168.204.118@tcp [ 7863.333723] Key type .llcrypt unregistered [ 7863.335536] Key type ._llcrypt unregistered [ 7897.294294] Key type ._llcrypt registered [ 7897.301325] Key type .llcrypt registered [ 7897.541311] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing set_hostid [ 7921.086887] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing load_modules_local [ 7922.296800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7922.485392] alg: No test for adler32 (adler32-zlib) [ 7923.545653] Lustre: Lustre: Build Version: 2.17.57_110_g063755c [ 7923.814345] LNet: Added LNI 192.168.204.118@tcp [8/256/0/180] [ 7925.504490] Key type lgssc registered [ 7926.728629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7959.322395] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7684 sec ========== 23:51:11 (1788839471) [ 7961.082341] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:51:12 (1788839472) === [ 7964.668669] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:51:16 (1788839476) === [ 8008.547994] Lustre: DEBUG MARKER: oleg418-server.virtnet: executing unload_modules_local [ 8011.177505] Key type lgssc unregistered [ 8011.562706] LNet: 242984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8011.574439] LNetError: 242984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8011.598058] LNet: Removed LNI 192.168.204.118@tcp [ 8012.643683] Key type .llcrypt unregistered [ 8012.646741] Key type ._llcrypt unregistered