[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 424618954 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002140] x2apic enabled [ 0.002902] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005622] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006008] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008080] LSM: Security Framework initializing [ 0.009005] Yama: becoming mindful. [ 0.009624] SELinux: Initializing. [ 0.010036] *** VALIDATE selinux *** [ 0.016634] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.020235] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.021113] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.022071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024068] *** VALIDATE tmpfs *** [ 0.025085] *** VALIDATE proc *** [ 0.025884] *** VALIDATE cgroup *** [ 0.026004] *** VALIDATE cgroup2 *** [ 0.027201] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030016] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.034127] debug: unmapping init [mem 0xffffffffa5859000-0xffffffffa5860fff] [ 0.036123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037413] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038009] ... version: 2 [ 0.038709] ... bit width: 48 [ 0.039005] ... generic registers: 4 [ 0.039681] ... value mask: 0000ffffffffffff [ 0.040005] ... max period: 00007fffffffffff [ 0.040914] ... fixed-purpose events: 3 [ 0.041005] ... event mask: 000000070000000f [ 0.042042] rcu: Hierarchical SRCU implementation. [ 0.043888] smp: Bringing up secondary CPUs ... [ 0.044337] x86: Booting SMP configuration: [ 0.045010] .... node #0, CPUs: #1 #2 #3 [ 0.047450] smp: Brought up 1 node, 4 CPUs [ 0.048721] smpboot: Max logical packages: 1 [ 0.049006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.236180] node 0 deferred pages initialised in 186ms [ 0.238130] devtmpfs: initialized [ 0.239164] x86/mm: Memory block size: 128MB [ 0.241125] gcov: version magic: 0x41383552 [ 0.242462] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243048] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244139] pinctrl core: initialized pinctrl subsystem [ 0.245087] [ 0.245382] ************************************************************* [ 0.246006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247005] ** ** [ 0.248005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249005] ** ** [ 0.250005] ** This means that this kernel is built to expose internal ** [ 0.251005] ** IOMMU data structures, which may compromise security on ** [ 0.252005] ** your system. ** [ 0.253005] ** ** [ 0.254005] ** If you see this message and you are not debugging the ** [ 0.255005] ** kernel, report this immediately to your vendor! ** [ 0.256005] ** ** [ 0.257005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258005] ************************************************************* [ 0.259402] NET: Registered protocol family 16 [ 0.260287] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.261024] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.262024] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263272] cpuidle: using governor menu [ 0.264434] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.265342] PCI: Using configuration type 1 for base access [ 0.267089] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.272106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.273021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.274106] cryptd: max_cpu_qlen set to 1000 [ 0.275751] ACPI: Added _OSI(Module Device) [ 0.277008] ACPI: Added _OSI(Processor Device) [ 0.277798] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.279009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.281548] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.285351] ACPI: Interpreter enabled [ 0.286035] ACPI: PM: (supports S0 S3 S4 S5) [ 0.286853] ACPI: Using IOAPIC for interrupt routing [ 0.288058] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.289263] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.295701] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.297018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.299009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.301031] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.303773] acpiphp: Slot [2] registered [ 0.304075] acpiphp: Slot [3] registered [ 0.304838] acpiphp: Slot [4] registered [ 0.306053] acpiphp: Slot [5] registered [ 0.306805] acpiphp: Slot [6] registered [ 0.307059] acpiphp: Slot [7] registered [ 0.307780] acpiphp: Slot [8] registered [ 0.308067] acpiphp: Slot [9] registered [ 0.308846] acpiphp: Slot [10] registered [ 0.310047] acpiphp: Slot [11] registered [ 0.310831] acpiphp: Slot [12] registered [ 0.312042] acpiphp: Slot [13] registered [ 0.312794] acpiphp: Slot [14] registered [ 0.313050] acpiphp: Slot [15] registered [ 0.313812] acpiphp: Slot [16] registered [ 0.315043] acpiphp: Slot [17] registered [ 0.315810] acpiphp: Slot [18] registered [ 0.316041] acpiphp: Slot [19] registered [ 0.316813] acpiphp: Slot [20] registered [ 0.318042] acpiphp: Slot [21] registered [ 0.318806] acpiphp: Slot [22] registered [ 0.319041] acpiphp: Slot [23] registered [ 0.319823] acpiphp: Slot [24] registered [ 0.321041] acpiphp: Slot [25] registered [ 0.321826] acpiphp: Slot [26] registered [ 0.322040] acpiphp: Slot [27] registered [ 0.322796] acpiphp: Slot [28] registered [ 0.324052] acpiphp: Slot [29] registered [ 0.325009] acpiphp: Slot [30] registered [ 0.325786] acpiphp: Slot [31] registered [ 0.326032] PCI host bridge to bus 0000:00 [ 0.326869] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332007] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.334012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.335059] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337604] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339520] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349034] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351006] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355470] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356491] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358018] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359360] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362963] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.371007] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.374795] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.378801] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.383011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.397013] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.403768] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.411008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.421010] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.426844] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.432008] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.435009] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.445019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.454394] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.458010] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.462009] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.470008] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.477607] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.481010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.485009] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.493008] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.499719] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.503008] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.506009] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.513985] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.522885] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.525396] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.528342] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.531415] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.534192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.539556] iommu: Default domain type: Passthrough [ 0.540276] SCSI subsystem initialized [ 0.540971] ACPI: bus type USB registered [ 0.542057] usbcore: registered new interface driver usbfs [ 0.543028] usbcore: registered new interface driver hub [ 0.544010] usbcore: registered new device driver usb [ 0.544960] pps_core: LinuxPPS API ver. 1 registered [ 0.546006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.547015] PTP clock support registered [ 0.548062] EDAC MC: Ver: 3.0.0 [ 0.549264] PCI: Using ACPI for IRQ routing [ 0.550579] NetLabel: Initializing [ 0.551008] NetLabel: domain hash size = 128 [ 0.551860] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.553037] NetLabel: unlabeled traffic allowed by default [ 0.555022] vgaarb: loaded [ 0.555610] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.556006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.562005] clocksource: Switched to clocksource kvm-clock [ 0.642095] VFS: Disk quotas dquot_6.6.0 [ 0.643019] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.644481] *** VALIDATE ramfs *** [ 0.645129] *** VALIDATE hugetlbfs *** [ 0.645966] pnp: PnP ACPI init [ 0.648045] pnp: PnP ACPI: found 6 devices [ 0.660804] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.662532] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.664080] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.666048] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.667348] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.668835] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.670534] NET: Registered protocol family 2 [ 0.672055] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.674982] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.677838] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.682082] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.683884] TCP: Hash tables configured (established 65536 bind 65536) [ 0.685415] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.686981] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.688613] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.690162] NET: Registered protocol family 1 [ 0.691920] RPC: Registered named UNIX socket transport module. [ 0.692965] RPC: Registered udp transport module. [ 0.693805] RPC: Registered tcp transport module. [ 0.694623] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.695802] NET: Registered protocol family 44 [ 0.696716] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.697800] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.698836] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.700775] PCI: CLS 0 bytes, default 64 [ 0.702158] Unpacking initramfs... [ 1.876457] debug: unmapping init [mem 0xffff8c697cc54000-0xffff8c697ffbffff] [ 1.878761] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.879908] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.881458] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.342474] Initialise system trusted keyrings [ 2.343429] Key type blacklist registered [ 2.344394] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.350084] zbud: loaded [ 2.351940] *** VALIDATE nfs *** [ 2.352598] *** VALIDATE nfs4 *** [ 2.353449] pstore: using deflate compression [ 2.355024] Platform Keyring initialized [ 2.418501] NET: Registered protocol family 38 [ 2.419886] Key type asymmetric registered [ 2.420861] Asymmetric key parser 'x509' registered [ 2.422209] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.424210] io scheduler mq-deadline registered [ 2.425280] io scheduler kyber registered [ 2.426388] io scheduler bfq registered [ 2.427286] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.428875] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.430484] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.432234] ACPI: Power Button [PWRF] [ 2.506505] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.578316] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.715428] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.825592] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.987579] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.015189] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.042162] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.045691] Non-volatile memory driver v1.3 [ 3.046823] Linux agpgart interface v0.103 [ 3.072939] virtio_blk virtio1: [vda] 132880 512-byte logical blocks (68.0 MB/64.9 MiB) [ 3.075428] vda: detected capacity change from 0 to 68034560 [ 3.086744] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.089457] vdb: detected capacity change from 0 to 1073741824 [ 3.101843] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.104600] vdc: detected capacity change from 0 to 2621440000 [ 3.115972] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.118506] vdd: detected capacity change from 0 to 2621440000 [ 3.130323] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.133051] vde: detected capacity change from 0 to 4294967296 [ 3.144420] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.145845] vdf: detected capacity change from 0 to 4294967296 [ 3.149700] libphy: Fixed MDIO Bus: probed [ 3.153494] usbcore: registered new interface driver usbserial_generic [ 3.155645] usbserial: USB Serial support registered for generic [ 3.157446] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.160043] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.161302] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.163487] mousedev: PS/2 mouse device common for all mice [ 3.165691] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.167800] rtc_cmos 00:05: RTC can wake from S4 [ 3.169745] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.170496] rtc_cmos 00:05: registered as rtc0 [ 3.173372] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.174882] intel_pstate: CPU model not supported [ 3.176855] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.178063] hid: raw HID events driver (C) Jiri Kosina [ 3.179946] usbcore: registered new interface driver usbhid [ 3.181015] usbhid: USB HID core driver [ 3.181824] drop_monitor: Initializing network drop monitor service [ 3.183113] Initializing XFRM netlink socket [ 3.184264] NET: Registered protocol family 10 [ 3.185956] Segment Routing with IPv6 [ 3.186784] NET: Registered protocol family 17 [ 3.187960] mpls_gso: MPLS GSO support [ 3.192565] RAS: Correctable Errors collector initialized. [ 3.193711] AVX version of gcm_enc/dec engaged. [ 3.194595] AES CTR mode by8 optimization enabled [ 3.240298] sched_clock: Marking stable (3240270345, 0)->(3827240936, -586970591) [ 3.242120] registered taskstats version 1 [ 3.243440] Loading compiled-in X.509 certificates [ 3.244440] zswap: loaded using pool lzo/zbud [ 3.262387] Key type big_key registered [ 3.269109] Key type encrypted registered [ 3.270633] ima: No TPM chip found, activating TPM-bypass! [ 3.272597] ima: Allocated hash algorithm: sha1 [ 3.274203] ima: No architecture policies found [ 3.275881] evm: Initialising EVM extended attributes: [ 3.277643] evm: security.selinux [ 3.278792] evm: security.ima [ 3.279847] evm: security.capability [ 3.281101] evm: HMAC attrs: 0x1 [ 3.283278] rtc_cmos 00:05: setting system clock to 2025-08-20 13:10:23 UTC (1755695423) [ 3.289707] debug: unmapping init [mem 0xffffffffa6803000-0xffffffffa69fffff] [ 3.292457] debug: unmapping init [mem 0xffffffffa5582000-0xffffffffa5858fff] [ 3.301071] Write protecting the kernel read-only data: 28672k [ 3.302980] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 3.304381] debug: unmapping init [mem 0xffffffffa4514000-0xffffffffa45fffff] [ 3.326390] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.331153] systemd[1]: Detected virtualization kvm. [ 3.333222] systemd[1]: Detected architecture x86-64. [ 3.335066] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.362443] systemd[1]: No hostname configured. [ 3.363297] systemd[1]: Set hostname to . [ 3.364311] random: systemd: uninitialized urandom read (16 bytes read) [ 3.365544] systemd[1]: Initializing machine ID from random generator. [ 3.394614] random: ln: uninitialized urandom read (6 bytes read) [ 3.466481] random: systemd: uninitialized urandom read (16 bytes read) [ 3.467842] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.470409] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.473542] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.996594] device-mapper: uevent: version 1.0.3 [ 3.997765] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.455885] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.657414] virtio_net virtio0 ens2: renamed from eth0 [ 4.746244] scsi host0: ata_piix [ 4.809406] scsi host1: ata_piix [ 4.810507] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.811910] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.322907] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.785306] random: crng init done [ 9.786131] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.915486] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.803587] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.977763] SELinux: Disabled at runtime. [ 11.033415] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.041394] systemd[1]: Detected virtualization kvm. [ 11.043218] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.433528] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.436688] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.440548] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.444253] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.447333] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.454487] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.458920] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached targ[ 11.481636] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.818303] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.140414] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.269941] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.300964] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.327222] EDAC sbridge: Ver: 1.1.2 [ 13.338113] Key type dns_resolver registered [ 13.563556] NFS: Registering the id_resolver key type [ 13.564614] Key type id_resolver registered [ 13.565426] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg253-server login: [ 37.716578] libcfs: loading out-of-tree module taints kernel. [ 37.725887] Key type ._llcrypt registered [ 37.727357] Key type .llcrypt registered [ 37.761334] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 45.165160] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 45.639296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 45.645370] alg: No test for adler32 (adler32-zlib) [ 46.617945] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 46.890163] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 48.511132] Key type lgssc registered [ 49.039981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.150724] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 56.063322] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.705091] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 62.646767] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.262836] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.561623] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 75.394707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.418360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.429518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.519578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.530764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.570363] Lustre: lustre-MDT0000: new disk, initializing [ 76.598586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.606475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.983319] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.291844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.333800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 84.373948] Lustre: 6470:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 84.388835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 84.391454] Lustre: Skipped 1 previous similar message [ 84.440843] Lustre: lustre-MDT0001: new disk, initializing [ 84.467322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 84.480711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 84.488859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 85.747611] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.316627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 92.516773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 92.561675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 92.690106] Lustre: lustre-OST0000: new disk, initializing [ 92.692356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 92.714138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.877417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 93.882438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 93.919902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 95.185791] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.852413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.876720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 101.912131] Lustre: lustre-OST0001: new disk, initializing [ 101.914878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 101.933400] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 104.011321] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.558824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 109.564429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 109.582178] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 110.665767] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 113.785902] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 121.113642] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing check_logdir /tmp/testlogs/ [ 122.859510] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing yml_node [ 124.862259] Lustre: DEBUG MARKER: Client: 2.16.57.97 [ 126.103593] Lustre: DEBUG MARKER: MDS: 2.16.57.97 [ 127.353463] Lustre: DEBUG MARKER: OSS: 2.16.57.97 [ 127.991796] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 20 09:12:27 EDT 2025 [ 135.949340] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 136.598323] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 140.255914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 140.257274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 140.258916] Lustre: Skipped 3 previous similar messages [ 140.263727] Lustre: Skipped 2 previous similar messages [ 144.265314] LustreError: 12115:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 144.303654] Lustre: server umount lustre-MDT0000 complete [ 145.674737] LustreError: 6462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695565 with bad export cookie 10767147924870419367 [ 145.676301] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 145.677814] LustreError: 6462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 150.495670] LustreError: 6477:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 150.495845] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 150.496174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 150.499737] LustreError: 6477:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 150.507432] Lustre: Skipped 1 previous similar message [ 151.802516] LustreError: 12316:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 151.805081] LustreError: 12316:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 151.890697] Lustre: server umount lustre-MDT0001 complete [ 153.458129] LustreError: 12519:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 153.462203] LustreError: 12519:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 153.491396] Lustre: server umount lustre-OST0000 complete [ 155.136314] Lustre: server umount lustre-OST0001 complete [ 157.591213] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 160.862688] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 164.697991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.555938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.187217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.176881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.195434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.235414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.366563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.388889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.431437] Lustre: lustre-MDT0000: new disk, initializing [ 176.461342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.469829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.845256] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 183.215902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.255708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 183.313395] Lustre: 14887:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.476906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.221090] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 189.230689] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 190.691590] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 193.682528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 193.720315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 193.861838] Lustre: lustre-OST0000: new disk, initializing [ 193.865641] Lustre: Skipped 1 previous similar message [ 193.869593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.874929] Lustre: Skipped 2 previous similar messages [ 193.920615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.924474] Lustre: Skipped 1 previous similar message [ 195.523091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.526388] Lustre: Skipped 1 previous similar message [ 195.529142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.573169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.360591] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 200.273631] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 201.709763] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 201.802967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 203.362539] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 203.468803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 205.795954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 205.796918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 205.800486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 205.808964] Lustre: Skipped 1 previous similar message [ 205.811791] Lustre: Skipped 1 previous similar message [ 210.286488] LustreError: 17796:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 210.288810] LustreError: 17796:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 210.356880] Lustre: server umount lustre-OST0000 complete [ 214.496802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 214.502107] LustreError: Skipped 1 previous similar message [ 214.505107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 214.514789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 214.518969] Lustre: Skipped 1 previous similar message [ 221.152730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 221.155766] Lustre: Skipped 3 previous similar messages [ 226.271159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 226.305811] LustreError: 17997:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 226.307267] LustreError: 17997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 226.336650] Lustre: server umount lustre-MDT0000 complete [ 227.766277] LustreError: 14877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695647 with bad export cookie 10767147924870420886 [ 227.769083] LustreError: 14877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 227.769602] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 227.907652] Lustre: server umount lustre-MDT0001 complete [ 229.525221] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:14:08 (1755695648) [ 232.516266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.690094] LustreError: 18724:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.720576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.336782] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 238.048568] LustreError: 18724:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 238.393359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 240.267361] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 242.908755] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 244.665925] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 247.935277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 250.699456] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 253.274423] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 262.631290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.632780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 262.636927] Lustre: Skipped 1 previous similar message [ 262.642200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 262.646378] Lustre: Skipped 3 previous similar messages [ 263.663761] LustreError: 20952:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 263.666321] LustreError: 20952:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 263.745697] Lustre: server umount lustre-OST0000 complete [ 267.701933] Lustre: server umount lustre-MDT0000 complete [ 267.748863] LustreError: 18724:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 267.757882] LustreError: 18724:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 269.456872] LustreError: 18704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695689 with bad export cookie 10767147924870421761 [ 269.462777] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.465995] LustreError: 18704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 269.724412] Lustre: server umount lustre-MDT0001 complete [ 274.648853] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 276.018636] Key type lgssc unregistered [ 276.181506] LNet: 21881:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.185111] LNetError: 21881:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 276.192572] LNet: Removed LNI 192.168.202.153@tcp [ 276.688175] Key type .llcrypt unregistered [ 276.690292] Key type ._llcrypt unregistered [ 286.932404] Key type ._llcrypt registered [ 286.933863] Key type .llcrypt registered [ 286.990294] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 287.479777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 287.496309] alg: No test for adler32 (adler32-zlib) [ 288.362370] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 288.458771] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 290.047181] Key type lgssc registered [ 290.679233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 294.645366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 294.660139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 295.848307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.466580] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 301.198900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 301.400548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 302.961856] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 305.573460] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 307.413654] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 310.848704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 311.015546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 313.282359] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 315.606364] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 321.268724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 326.628040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.629715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 326.629943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 326.631544] Lustre: Skipped 1 previous similar message [ 331.749323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 331.753021] Lustre: Skipped 2 previous similar messages [ 332.531134] LustreError: 25914:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 332.619367] Lustre: server umount lustre-OST0000 complete [ 336.867557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.870172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.876096] Lustre: Skipped 1 previous similar message [ 336.884482] Lustre: Skipped 1 previous similar message [ 340.626635] LustreError: 26118:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 340.631152] LustreError: 26118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 340.673166] Lustre: server umount lustre-MDT0000 complete [ 341.985434] LustreError: 23243:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.000386] LustreError: 23243:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 342.525224] LustreError: 23225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695762 with bad export cookie 12037646380738086012 [ 342.530871] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.533064] LustreError: 23225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 342.626503] LustreError: 26321:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 342.630830] LustreError: 26321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 342.726671] Lustre: server umount lustre-MDT0001 complete [ 347.139849] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 348.472979] Key type lgssc unregistered [ 348.616461] LNet: 26847:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.619409] LNetError: 26847:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.632379] LNet: Removed LNI 192.168.202.153@tcp [ 349.114928] Key type .llcrypt unregistered [ 349.116952] Key type ._llcrypt unregistered [ 360.366589] Key type ._llcrypt registered [ 360.367948] Key type .llcrypt registered [ 360.410991] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 360.788631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.811059] alg: No test for adler32 (adler32-zlib) [ 361.651168] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 361.723353] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 363.295170] Key type lgssc registered [ 363.850389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 366.669871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 366.682947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.886478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.154098] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 372.879109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 373.059297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.633598] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 377.010720] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 378.552754] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 384.747408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 384.884524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.275724] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 389.739781] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 393.069229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 403.430721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.430810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 403.444634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 403.447254] LustreError: Skipped 1 previous similar message [ 403.455953] Lustre: Skipped 1 previous similar message [ 405.042160] LustreError: 30805:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 405.133721] Lustre: server umount lustre-OST0000 complete [ 408.546264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.553225] Lustre: Skipped 1 previous similar message [ 408.556760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 408.560379] Lustre: Skipped 1 previous similar message [ 413.664079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 413.665895] Lustre: Skipped 1 previous similar message [ 418.784933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 418.788558] Lustre: Skipped 1 previous similar message [ 421.343130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.399742] LustreError: 31008:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 421.403402] LustreError: 31008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 421.433780] Lustre: server umount lustre-MDT0000 complete [ 422.880810] LustreError: 28211:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.887247] LustreError: 28211:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 422.893018] LustreError: 28193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695843 with bad export cookie 8715438566403394849 [ 422.895687] LustreError: 28193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 422.896291] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 422.962892] LustreError: 31210:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 422.964828] LustreError: 31210:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 423.038205] Lustre: server umount lustre-MDT0001 complete [ 426.302758] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 427.237932] Key type lgssc unregistered [ 427.355299] LNet: 31736:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.357421] LNetError: 31736:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.367306] LNet: Removed LNI 192.168.202.153@tcp [ 427.643236] Key type .llcrypt unregistered [ 427.644705] Key type ._llcrypt unregistered [ 438.248410] Key type ._llcrypt registered [ 438.250084] Key type .llcrypt registered [ 438.297454] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 438.710294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 438.717680] alg: No test for adler32 (adler32-zlib) [ 439.578431] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 439.654388] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 441.239179] Key type lgssc registered [ 441.834547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 445.328815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 445.340096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.536240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.111839] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 452.087037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 452.265601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 453.680948] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 455.861529] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 457.440073] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 460.538436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 460.732602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.114371] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 465.620970] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 470.958165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 476.133809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.134173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.139294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.145435] Lustre: Skipped 1 previous similar message [ 476.151445] Lustre: Skipped 1 previous similar message [ 480.175232] LustreError: 35327:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 480.266303] Lustre: server umount lustre-OST0000 complete [ 483.296787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.301427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.309659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.369917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.370864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.059309] LustreError: 35529:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 488.061127] LustreError: 35529:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 488.086586] Lustre: server umount lustre-MDT0000 complete [ 489.466433] LustreError: 33081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695909 with bad export cookie 9011592535770633054 [ 489.470362] LustreError: 33081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 489.470510] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.540936] LustreError: 35730:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 489.543014] LustreError: 35730:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 489.620707] Lustre: server umount lustre-MDT0001 complete [ 493.530221] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 494.820228] Key type lgssc unregistered [ 494.956574] LNet: 36257:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.959645] LNetError: 36257:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 494.968362] LNet: Removed LNI 192.168.202.153@tcp [ 495.232603] Key type .llcrypt unregistered [ 495.233715] Key type ._llcrypt unregistered [ 506.183524] Key type ._llcrypt registered [ 506.184705] Key type .llcrypt registered [ 506.218415] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 506.520257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 506.552130] alg: No test for adler32 (adler32-zlib) [ 507.407281] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 507.490928] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 509.079182] Key type lgssc registered [ 509.651944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 513.228402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 513.237909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.494268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.308267] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 520.350523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 520.484917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 522.291708] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 524.384279] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 525.908184] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 529.285205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 529.484146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.905235] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 534.439685] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 539.763258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 549.954488] Lustre: lustre-OST0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 550.374589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.379289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.586050] LustreError: 39873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 553.690924] Lustre: server umount lustre-OST0000 complete [ 560.610381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.611346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 560.617827] Lustre: Skipped 2 previous similar messages [ 560.621491] Lustre: Skipped 2 previous similar messages [ 565.729062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 565.732778] Lustre: Skipped 1 previous similar message [ 569.825446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.829388] Lustre: Skipped 1 previous similar message [ 571.359099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.403439] LustreError: 40226:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 571.407148] LustreError: 40226:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 571.436843] Lustre: server umount lustre-MDT0000 complete [ 572.882062] LustreError: 37600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695993 with bad export cookie 16866221981503911842 [ 572.884911] LustreError: 37600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 572.885049] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.231161] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 587.287190] LustreError: 40429:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 587.290940] LustreError: 40429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 587.381208] Lustre: server umount lustre-MDT0001 complete [ 591.609573] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 592.995072] Key type lgssc unregistered [ 593.142498] LNet: 40957:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.146425] LNetError: 40957:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.159565] LNet: Removed LNI 192.168.202.153@tcp [ 593.566854] Key type .llcrypt unregistered [ 593.568576] Key type ._llcrypt unregistered [ 604.125738] Key type ._llcrypt registered [ 604.127467] Key type .llcrypt registered [ 604.171789] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 604.493512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 604.509458] alg: No test for adler32 (adler32-zlib) [ 605.363266] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 605.461447] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 607.047177] Key type lgssc registered [ 607.636550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 610.975381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 610.980905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.160787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 613.824527] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 617.423336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 617.544595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 619.153499] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 621.515332] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 622.904395] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 626.200682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 626.413728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.827656] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 630.978935] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 638.700885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 646.719749] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.53@tcp (stopping) [ 648.162393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 648.167418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.179219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.189937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.192249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.200802] Lustre: Skipped 1 previous similar message [ 650.445420] LustreError: 44573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 650.495621] Lustre: server umount lustre-MDT0000 complete [ 651.825134] LustreError: 42319:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.829152] LustreError: 42319:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 652.059285] LustreError: 43035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696072 with bad export cookie 13923833638414459916 [ 652.064335] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.304622] LustreError: 42347:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.304909] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.312693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 654.314468] Lustre: Skipped 2 previous similar messages [ 656.946602] LustreError: 42347:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.189587] LustreError: 44776:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 658.192948] LustreError: 44776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 658.308266] Lustre: server umount lustre-MDT0001 complete [ 659.277737] LustreError: 42303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.53@tcp arrived at 1755696079 with bad export cookie 13923833638414460567 [ 659.285913] LustreError: 42303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 672.305800] LustreError: 44979:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 672.311086] LustreError: 44979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 672.424253] Lustre: server umount lustre-OST0000 complete [ 679.776654] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 681.082559] Key type lgssc unregistered [ 681.257667] LNet: 45811:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.262302] LNetError: 45811:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.270626] LNet: Removed LNI 192.168.202.153@tcp [ 681.767267] Key type .llcrypt unregistered [ 681.769891] Key type ._llcrypt unregistered [ 693.924952] Key type ._llcrypt registered [ 693.926759] Key type .llcrypt registered [ 693.976722] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 694.302749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.313528] alg: No test for adler32 (adler32-zlib) [ 695.188034] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 695.289323] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 696.871160] Key type lgssc registered [ 697.277979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.250214] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 700.262355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 717.791414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 723.935242] LustreError: 47150:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 753.631248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.353777] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 758.885798] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 923.360857] LustreError: 47853:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 923.457212] Lustre: server umount lustre-OST0000 complete [ 931.116945] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 932.609472] Key type lgssc unregistered [ 932.784681] LNet: 48683:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.789443] LNetError: 48683:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.799600] LNet: Removed LNI 192.168.202.153@tcp [ 933.245728] Key type .llcrypt unregistered [ 933.247811] Key type ._llcrypt unregistered [ 944.129564] Key type ._llcrypt registered [ 944.131066] Key type .llcrypt registered [ 944.173920] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 944.477860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.525253] alg: No test for adler32 (adler32-zlib) [ 945.370227] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 945.470630] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 947.055101] Key type lgssc registered [ 947.606870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 950.963347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 950.979157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.193660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.781671] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 957.758263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 957.895359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.582382] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 962.152350] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 963.993548] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 967.302965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 967.486948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.992591] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 972.387331] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.795514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.795547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.807245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.407246] LustreError: 52275:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 975.501764] Lustre: server umount lustre-OST0000 complete [ 978.400556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 978.405122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.413117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.985153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.986238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.372700] LustreError: 52477:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 983.376514] LustreError: 52477:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 983.410900] Lustre: server umount lustre-MDT0000 complete [ 985.182699] LustreError: 50027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696405 with bad export cookie 1716272000821396217 [ 985.186605] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.188265] LustreError: 50027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 985.272920] LustreError: 52681:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 985.276243] LustreError: 52681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 985.363128] Lustre: server umount lustre-MDT0001 complete [ 990.049800] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 991.518119] Key type lgssc unregistered [ 991.671629] LNet: 53206:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.676248] LNetError: 53206:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.685642] LNet: Removed LNI 192.168.202.153@tcp [ 992.132744] Key type .llcrypt unregistered [ 992.134460] Key type ._llcrypt unregistered [ 1002.907718] Key type ._llcrypt registered [ 1002.909292] Key type .llcrypt registered [ 1002.956716] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1003.518802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.531747] alg: No test for adler32 (adler32-zlib) [ 1004.425867] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1004.550402] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1006.167176] Key type lgssc registered [ 1006.805726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.785988] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.796891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1027.295491] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1033.439367] LustreError: 54543:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 1063.135224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.588545] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1067.878282] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1071.126981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.393767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.421924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1072.985904] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1076.708306] LustreError: 54570:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.716706] LustreError: 54570:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1077.068002] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1077.286830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.026188] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1081.501754] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1083.188784] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1086.944787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.949078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.956800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.458429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1087.462951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.470937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1088.488534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1088.492027] Lustre: Skipped 1 previous similar message [ 1090.225449] LustreError: 56803:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1090.324022] Lustre: server umount lustre-OST0000 complete [ 1097.696383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1097.700927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.709037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1101.067683] LustreError: 57161:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1101.071631] LustreError: 57161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1101.102480] Lustre: server umount lustre-MDT0000 complete [ 1102.593083] LustreError: 54550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696522 with bad export cookie 17273586140959586433 [ 1102.599151] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.599681] LustreError: 54550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1108.746597] LustreError: 57363:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1108.750428] LustreError: 57363:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1108.848068] Lustre: server umount lustre-MDT0001 complete [ 1112.937572] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1114.446615] Key type lgssc unregistered [ 1114.600499] LNet: 57891:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.602610] LNetError: 57891:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.612429] LNet: Removed LNI 192.168.202.153@tcp [ 1114.925338] Key type .llcrypt unregistered [ 1114.926323] Key type ._llcrypt unregistered [ 1125.278298] Key type ._llcrypt registered [ 1125.279537] Key type .llcrypt registered [ 1125.320741] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1125.615524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1125.621239] alg: No test for adler32 (adler32-zlib) [ 1126.454379] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1126.524893] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1128.095199] Key type lgssc registered [ 1128.669815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1132.150430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1132.162710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.325478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.771247] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1138.799894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1138.995977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.687748] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1143.166163] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.822808] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1147.505944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1147.617476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.850952] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1152.297977] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1157.869643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1165.280333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.284892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.292860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.358470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.358761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.370281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.186165] LustreError: 61503:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1171.278097] Lustre: server umount lustre-OST0000 complete [ 1173.474544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.475649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.481729] Lustre: Skipped 1 previous similar message [ 1173.485583] Lustre: Skipped 1 previous similar message [ 1178.593045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.596894] Lustre: Skipped 1 previous similar message [ 1183.712957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1183.716553] Lustre: Skipped 1 previous similar message [ 1187.295132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1187.348644] LustreError: 61705:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1187.352321] LustreError: 61705:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1187.387787] Lustre: server umount lustre-MDT0000 complete [ 1188.832689] LustreError: 60374:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.842278] LustreError: 60374:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1189.130197] LustreError: 59234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696609 with bad export cookie 2615215925239233510 [ 1189.135838] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1189.137083] LustreError: 59234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1189.257174] LustreError: 61908:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1189.260889] LustreError: 61908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1189.347210] Lustre: server umount lustre-MDT0001 complete [ 1194.077338] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1195.491565] Key type lgssc unregistered [ 1195.653673] LNet: 62435:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.658274] LNetError: 62435:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.668582] LNet: Removed LNI 192.168.202.153@tcp [ 1196.132233] Key type .llcrypt unregistered [ 1196.134288] Key type ._llcrypt unregistered [ 1212.263519] Key type ._llcrypt registered [ 1212.265233] Key type .llcrypt registered [ 1212.316541] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1212.636741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.644643] alg: No test for adler32 (adler32-zlib) [ 1213.513830] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1213.613895] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1215.215189] Key type lgssc registered [ 1215.843099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.695995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.707849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.950662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.755287] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1227.008960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1227.216626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.972177] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1231.518212] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1233.240290] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1236.473601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1236.658210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1239.114706] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1241.654627] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1245.867914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1249.656662] Lustre: Failing over lustre-MDT0000 [ 1249.904048] LustreError: 66321:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1249.931908] Lustre: server umount lustre-MDT0000 complete [ 1251.299679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.301836] LustreError: 64085:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.307297] Lustre: Skipped 2 previous similar messages [ 1251.316336] LustreError: 64085:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1252.415771] LustreError: 66230:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.425509] LustreError: 66230:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1254.329387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.396200] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.470292] Lustre: *** cfs_fail_loc=135, val=0*** [ 1254.509122] LustreError: 66831:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000 failed (-2) [ 1254.511311] LustreError: 66831:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -2 [ 1254.513775] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1254.513775] [ 1254.516657] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1254.520956] LustreError: 66819:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1254.523413] LustreError: 66819:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 1254.526170] LustreError: 66819:0:(obd_config.c:832:class_cleanup()) Device 5 not setup [ 1254.534813] Lustre: server umount lustre-MDT0000 complete [ 1254.536094] LustreError: 66819:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1256.417761] LustreError: 66234:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.427189] LustreError: 66234:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1257.295044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.359341] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.508806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.560481] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1259.212141] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1262.561229] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1262.565740] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1262.647535] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1262.661731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1267.686092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.688186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1267.692938] Lustre: Skipped 1 previous similar message [ 1269.743172] LustreError: 67631:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1269.747057] LustreError: 67631:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1269.840138] Lustre: server umount lustre-OST0000 complete [ 1272.802041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1272.802425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.803612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.803620] Lustre: Skipped 1 previous similar message [ 1272.817220] Lustre: Skipped 1 previous similar message [ 1277.705750] LustreError: 67833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1277.709529] LustreError: 67833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1277.743917] Lustre: server umount lustre-MDT0000 complete [ 1277.920967] LustreError: 64090:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.930343] LustreError: 64090:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1279.339981] LustreError: 64804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696699 with bad export cookie 1126683482566643650 [ 1279.346447] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1279.346616] LustreError: 64804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1279.532310] Lustre: server umount lustre-MDT0001 complete [ 1283.906671] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1285.229623] Key type lgssc unregistered [ 1285.388633] LNet: 68561:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.392299] LNetError: 68561:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.404445] LNet: Removed LNI 192.168.202.153@tcp [ 1285.849313] Key type .llcrypt unregistered [ 1285.851254] Key type ._llcrypt unregistered [ 1297.237769] Key type ._llcrypt registered [ 1297.239728] Key type .llcrypt registered [ 1297.303697] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1297.698213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.742982] alg: No test for adler32 (adler32-zlib) [ 1298.663861] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1298.773754] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1300.383159] Key type lgssc registered [ 1300.767456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.045530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1304.052541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.236048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.843032] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1310.749708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1310.936865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1312.623608] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1314.982992] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1316.241564] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1319.431430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1319.596282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.042709] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1324.458635] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1329.836937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1337.374975] Lustre: Failing over lustre-MDT0000 [ 1337.527365] LustreError: 72178:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1337.560365] Lustre: server umount lustre-MDT0000 complete [ 1339.962412] LustreError: 69922:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.968851] LustreError: 69922:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1340.385480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.392769] Lustre: Skipped 2 previous similar messages [ 1341.527090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.590557] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.646888] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1341.708552] LustreError: 72685:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000 failed (-9) [ 1341.711610] LustreError: 72685:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -9 [ 1341.714271] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1341.714271] [ 1341.717116] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1341.721165] LustreError: 72673:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1341.723472] LustreError: 72673:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -9 [ 1341.726101] LustreError: 72673:0:(obd_config.c:832:class_cleanup()) Device 5 not setup [ 1341.734493] Lustre: server umount lustre-MDT0000 complete [ 1341.735662] LustreError: 72673:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1344.555599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.621521] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.773404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.817808] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1345.075087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1346.507662] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1350.121992] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1350.132727] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1350.163520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1352.672372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1352.676847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.684808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.234195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1355.238294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.239585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.248464] Lustre: Skipped 1 previous similar message [ 1357.554161] LustreError: 73490:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1357.558083] LustreError: 73490:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1357.651655] Lustre: server umount lustre-OST0000 complete [ 1360.355134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.355313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.356156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.362412] Lustre: Skipped 1 previous similar message [ 1365.473041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1365.476620] Lustre: Skipped 2 previous similar messages [ 1365.578570] LustreError: 73692:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1365.582398] LustreError: 73692:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1365.617205] Lustre: server umount lustre-MDT0000 complete [ 1367.333958] LustreError: 69905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696787 with bad export cookie 8764386867289063919 [ 1367.336126] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1367.336970] LustreError: 69905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1367.466849] Lustre: server umount lustre-MDT0001 complete [ 1370.818414] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1371.914049] Key type lgssc unregistered [ 1372.069449] LNet: 74420:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.071652] LNetError: 74420:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.079458] LNet: Removed LNI 192.168.202.153@tcp [ 1372.446944] Key type .llcrypt unregistered [ 1372.449131] Key type ._llcrypt unregistered [ 1376.867332] Key type ._llcrypt registered [ 1376.868897] Key type .llcrypt registered [ 1376.906335] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1377.011202] Key type .llcrypt unregistered [ 1377.012761] Key type ._llcrypt unregistered [ 1386.952462] Key type ._llcrypt registered [ 1386.953977] Key type .llcrypt registered [ 1386.996913] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1387.532714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1387.540061] alg: No test for adler32 (adler32-zlib) [ 1388.404277] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1388.492747] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1390.079133] Key type lgssc registered [ 1390.522946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.374886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.382769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.528871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.794143] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1398.287749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1398.416936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1399.639569] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1401.595767] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.743185] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1404.893654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1405.004712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.887418] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1408.572443] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1420.393943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1428.959892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1428.962261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.965838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.011101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1431.011315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.011801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.133427] LustreError: 78498:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1433.191295] Lustre: server umount lustre-OST0000 complete [ 1434.591985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1434.594533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.598595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.600515] Lustre: Skipped 1 previous similar message [ 1440.571222] LustreError: 78699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1440.572997] LustreError: 78699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1440.598803] Lustre: server umount lustre-MDT0000 complete [ 1441.248063] LustreError: 76952:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.252021] LustreError: 76952:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1441.789284] LustreError: 76228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696862 with bad export cookie 3285126371271950094 [ 1441.792072] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.795716] LustreError: 76228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1441.868596] LustreError: 78900:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1441.870206] LustreError: 78900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1441.939153] Lustre: server umount lustre-MDT0001 complete [ 1445.542838] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1446.743784] Key type lgssc unregistered [ 1446.881407] LNet: 79428:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.885455] LNetError: 79428:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.893314] LNet: Removed LNI 192.168.202.153@tcp [ 1447.204839] Key type .llcrypt unregistered [ 1447.205858] Key type ._llcrypt unregistered [ 1455.821971] Key type ._llcrypt registered [ 1455.823706] Key type .llcrypt registered [ 1455.868879] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1456.242420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1456.249395] alg: No test for adler32 (adler32-zlib) [ 1457.120458] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1457.201171] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1458.791195] Key type lgssc registered [ 1459.140162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.908931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.917973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.104460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.506462] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1467.681642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1467.839497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1469.366349] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1471.734656] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1473.096795] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1475.435535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1475.559392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.461417] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1479.285815] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1487.846825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1492.963097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.964049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.965594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.971697] Lustre: Skipped 1 previous similar message [ 1494.321205] LustreError: 83037:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1494.419223] Lustre: server umount lustre-OST0000 complete [ 1497.790528] LustreError: 83240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1497.792482] LustreError: 83240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1497.820450] Lustre: server umount lustre-MDT0000 complete [ 1498.081870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.086803] LustreError: 80789:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.094114] LustreError: 80789:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1499.030174] LustreError: 83242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696919 with bad export cookie 2274324572206534240 [ 1499.032888] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.033124] LustreError: 83242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1499.103577] LustreError: 83442:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1499.105296] LustreError: 83442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1499.181347] Lustre: server umount lustre-MDT0001 complete [ 1502.866198] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1503.938429] Key type lgssc unregistered [ 1504.068486] LNet: 83968:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.071576] LNetError: 83968:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.078321] LNet: Removed LNI 192.168.202.153@tcp [ 1504.392396] Key type .llcrypt unregistered [ 1504.393399] Key type ._llcrypt unregistered [ 1513.444840] Key type ._llcrypt registered [ 1513.445933] Key type .llcrypt registered [ 1513.489754] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1513.848893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.866924] alg: No test for adler32 (adler32-zlib) [ 1514.720125] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1514.805389] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1516.383351] Key type lgssc registered [ 1516.801156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1519.866488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1519.876382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.069151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.495468] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1526.084981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1526.238678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1527.571691] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1529.564593] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1530.829737] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1533.321619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1533.438711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.510610] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1537.375387] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1546.788042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1551.846031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1551.846335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.850565] LustreError: Skipped 1 previous similar message [ 1551.851966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1551.857765] Lustre: Skipped 1 previous similar message [ 1551.865807] Lustre: Skipped 1 previous similar message [ 1556.965477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.969246] Lustre: Skipped 1 previous similar message [ 1557.294148] LustreError: 87581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1557.371218] Lustre: server umount lustre-OST0000 complete [ 1562.082514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1562.083227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.095828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.099389] Lustre: Skipped 1 previous similar message [ 1567.200026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1567.201926] Lustre: Skipped 1 previous similar message [ 1572.321124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.324871] Lustre: Skipped 1 previous similar message [ 1573.343116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1573.377773] LustreError: 87783:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1573.379645] LustreError: 87783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1573.409684] Lustre: server umount lustre-MDT0000 complete [ 1574.760860] LustreError: 86939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696994 with bad export cookie 6353252164141283780 [ 1574.766954] LustreError: 86939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1574.768600] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1574.851602] LustreError: 87986:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1574.853915] LustreError: 87986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1574.953094] Lustre: server umount lustre-MDT0001 complete [ 1579.219197] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1580.299742] Key type lgssc unregistered [ 1580.442629] LNet: 88512:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.447167] LNetError: 88512:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.458562] LNet: Removed LNI 192.168.202.153@tcp [ 1580.929362] Key type .llcrypt unregistered [ 1580.931064] Key type ._llcrypt unregistered [ 1591.924330] Key type ._llcrypt registered [ 1591.925787] Key type .llcrypt registered [ 1591.968259] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1592.230338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.252183] alg: No test for adler32 (adler32-zlib) [ 1593.102185] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1593.199057] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1594.799178] Key type lgssc registered [ 1595.391681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1599.158313] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1599.170457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1616.735547] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1622.879403] LustreError: 89848:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 1653.599275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.972714] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1658.524181] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1661.253802] LustreError: 90702:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1661.339219] Lustre: server umount lustre-OST0000 complete [ 1665.480171] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:38:04 (1755697084) [ 1668.804775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.032420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.859820] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1674.209562] LustreError: 91371:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1674.218565] LustreError: 91371:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1675.102523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1675.310737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1677.095968] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1679.896375] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1681.700980] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1685.037858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1685.191468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.150774] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1688.932177] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1694.447132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1704.931459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.933925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1704.937530] Lustre: Skipped 1 previous similar message [ 1704.939185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1704.940993] Lustre: Skipped 1 previous similar message [ 1709.485477] LustreError: 93746:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1709.488562] LustreError: 93746:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1709.575248] Lustre: server umount lustre-OST0000 complete [ 1715.169199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.173896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.185581] Lustre: Skipped 1 previous similar message [ 1715.219328] Lustre: Skipped 1 previous similar message [ 1717.431451] LustreError: 93949:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1717.433483] LustreError: 93949:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1717.534381] Lustre: server umount lustre-MDT0000 complete [ 1720.255237] LustreError: 91351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697140 with bad export cookie 18437382808824257912 [ 1720.259268] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.269481] LustreError: 91351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1720.408586] LustreError: 94150:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1720.411084] LustreError: 94150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1720.747725] Lustre: server umount lustre-MDT0001 complete [ 1729.263182] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1730.445143] Key type lgssc unregistered [ 1730.582760] LNet: 94677:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.586435] LNetError: 94677:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.593476] LNet: Removed LNI 192.168.202.153@tcp [ 1730.932467] Key type .llcrypt unregistered [ 1730.933890] Key type ._llcrypt unregistered [ 1740.123036] Key type ._llcrypt registered [ 1740.123832] Key type .llcrypt registered [ 1740.159421] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1740.457645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.477049] alg: No test for adler32 (adler32-zlib) [ 1741.338907] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1741.422256] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1742.999157] Key type lgssc registered [ 1743.377344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1745.886271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1745.893994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1747.033263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.325833] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1751.372487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1751.517785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.813149] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1754.451588] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1755.650783] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1758.132290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1758.247252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.208630] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1762.026729] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1772.581837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1776.607991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.611423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.617400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.633332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1777.633716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.634539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.640682] Lustre: Skipped 1 previous similar message [ 1781.292113] LustreError: 98289:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1781.368229] Lustre: server umount lustre-OST0000 complete [ 1782.752265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.756471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.359860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.363327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.369617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.730937] LustreError: 98491:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1788.732343] LustreError: 98491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1788.759876] Lustre: server umount lustre-MDT0000 complete [ 1789.933714] LustreError: 98492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697210 with bad export cookie 13549577388397681637 [ 1789.937980] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1790.000637] LustreError: 98693:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1790.002564] LustreError: 98693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1790.090411] Lustre: server umount lustre-MDT0001 complete [ 1793.944638] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 1795.170528] Key type lgssc unregistered [ 1795.314411] LNet: 99220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.316546] LNetError: 99220:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.323529] LNet: Removed LNI 192.168.202.153@tcp [ 1795.684807] Key type .llcrypt unregistered [ 1795.686212] Key type ._llcrypt unregistered [ 1803.554468] Key type ._llcrypt registered [ 1803.555378] Key type .llcrypt registered [ 1803.604726] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1804.053616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.081102] alg: No test for adler32 (adler32-zlib) [ 1804.943968] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1805.034453] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 1806.615138] Key type lgssc registered [ 1806.994905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1809.763623] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1809.770328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1826.271438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1832.415167] LustreError: 100412:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 1862.111237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.235309] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1866.243602] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1868.676072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.777595] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1868.782172] LustreError: 101318:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1868.785135] LustreError: 101318:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 1868.789336] LustreError: 101318:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-MDT0000 [ 1868.852370] Lustre: server umount lustre-MDT0000 complete [ 1868.855065] LustreError: 101318:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1881.055712] LustreError: 101749:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1881.127389] Lustre: server umount lustre-OST0000 complete [ 1884.442567] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1887.597914] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1891.027237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.406700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1895.532871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1897.810201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1900.245452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1900.270681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.364996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1900.376782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1900.415240] Lustre: lustre-MDT0000: new disk, initializing [ 1900.439960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.446165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1901.821581] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1906.112794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1906.136825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1906.172538] Lustre: 104071:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.196298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1906.197947] Lustre: Skipped 1 previous similar message [ 1906.236376] Lustre: lustre-MDT0001: new disk, initializing [ 1906.269189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.285926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1906.289959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1907.695976] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1910.868655] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.139421] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1914.517882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1914.546130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1914.664634] Lustre: lustre-OST0000: new disk, initializing [ 1914.666473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1914.703422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.962087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1915.965333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1915.999101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1916.773300] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1919.983914] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1921.416283] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1921.497491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1922.926758] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1923.020492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1926.114836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.115432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1926.119536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1926.122736] LustreError: Skipped 1 previous similar message [ 1926.124715] Lustre: Skipped 1 previous similar message [ 1931.234165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.237454] Lustre: Skipped 1 previous similar message [ 1936.356530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1936.358671] Lustre: Skipped 1 previous similar message [ 1937.887172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1937.916105] LustreError: 106980:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1937.917900] LustreError: 106980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1937.994864] Lustre: server umount lustre-OST0000 complete [ 1941.472790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.473856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1941.476118] Lustre: Skipped 2 previous similar messages [ 1941.481122] Lustre: Skipped 1 previous similar message [ 1945.412345] LustreError: 107182:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1945.415408] LustreError: 107182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1945.449771] Lustre: server umount lustre-MDT0000 complete [ 1946.799542] LustreError: 104061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697367 with bad export cookie 6234339640006756302 [ 1946.803328] LustreError: 104061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1946.803509] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.970741] Lustre: server umount lustre-MDT0001 complete [ 1950.109467] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:42:49 (1755697369) [ 1951.255647] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1956.672880] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 1959.800719] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 1963.396431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1966.082179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.117964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1970.266819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.694192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1972.725306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.830345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1972.841868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1972.880330] Lustre: lustre-MDT0000: new disk, initializing [ 1972.907364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.916095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1974.314331] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1978.739594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1978.770479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1978.812268] Lustre: 110503:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1978.841382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1978.843443] Lustre: Skipped 1 previous similar message [ 1978.890703] Lustre: lustre-MDT0001: new disk, initializing [ 1978.942521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1978.945362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1980.319151] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1983.476497] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1984.746935] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1987.265235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1987.287994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1987.410336] Lustre: lustre-OST0000: new disk, initializing [ 1987.412441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1987.442733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.445306] Lustre: Skipped 1 previous similar message [ 1989.451311] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 1992.752621] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1994.329296] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1996.790264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1996.793363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1996.825963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1997.492980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1998.871512] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1998.951214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2001.889095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2001.891192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.892308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.894478] Lustre: Skipped 1 previous similar message [ 2001.899561] Lustre: Skipped 1 previous similar message [ 2005.676118] LustreError: 113432:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2005.677881] LustreError: 113432:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2005.746162] Lustre: server umount lustre-OST0000 complete [ 2007.008830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.012300] Lustre: Skipped 1 previous similar message [ 2012.128774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.130574] Lustre: Skipped 3 previous similar messages [ 2021.343268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2021.374754] LustreError: 113634:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2021.376604] LustreError: 113634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2021.402832] Lustre: server umount lustre-MDT0000 complete [ 2022.368737] LustreError: 110512:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.374141] LustreError: 110512:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2022.727327] LustreError: 110493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697442 with bad export cookie 6234339640006757170 [ 2022.731704] LustreError: 110493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2022.732792] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.894870] Lustre: server umount lustre-MDT0001 complete [ 2025.829240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.009785] LustreError: 114311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.039137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.396899] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2030.442926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2031.877774] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2033.743915] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2034.976138] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2037.223843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2039.368615] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2041.242234] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2059.298305] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2061.281026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2061.283348] LustreError: Skipped 1 previous similar message [ 2061.284563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.289382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2061.291307] Lustre: Skipped 2 previous similar messages [ 2066.418109] LustreError: 116657:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2066.419696] LustreError: 116657:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2066.490768] Lustre: server umount lustre-OST0000 complete [ 2071.519981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.522067] LustreError: Skipped 1 previous similar message [ 2071.523104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.525812] Lustre: Skipped 1 previous similar message [ 2082.271166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2082.338152] Lustre: server umount lustre-MDT0000 complete [ 2082.784117] LustreError: 115015:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.788697] LustreError: 115015:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2083.582162] LustreError: 115919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697503 with bad export cookie 6234339640006758038 [ 2083.584925] LustreError: 115919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2083.585846] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.649100] LustreError: 117060:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2083.651406] LustreError: 117060:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2083.742437] Lustre: server umount lustre-MDT0001 complete [ 2087.519729] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2088.695907] Key type lgssc unregistered [ 2088.840514] LNet: 117586:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.844382] LNetError: 117586:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.854330] LNet: Removed LNI 192.168.202.153@tcp [ 2089.177554] Key type .llcrypt unregistered [ 2089.179233] Key type ._llcrypt unregistered [ 2095.252260] Key type ._llcrypt registered [ 2095.253213] Key type .llcrypt registered [ 2095.303613] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 2101.465778] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2101.892885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.917695] alg: No test for adler32 (adler32-zlib) [ 2102.776652] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2102.871184] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2104.463121] Key type lgssc registered [ 2104.837759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.999326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2110.338690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2112.445494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2114.600917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2116.982908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2117.000919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2117.006354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.104182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2118.117339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2118.153231] Lustre: lustre-MDT0000: new disk, initializing [ 2118.180651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.189593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2119.470742] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2123.625089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2123.647383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2123.681635] Lustre: 120674:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2123.711544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2123.713988] Lustre: Skipped 1 previous similar message [ 2123.754690] Lustre: lustre-MDT0001: new disk, initializing [ 2123.790698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.805231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2123.808291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2125.151116] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2128.256569] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2129.554503] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2131.925215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2131.942807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2132.054661] Lustre: lustre-OST0000: new disk, initializing [ 2132.056622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2132.092619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.112969] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2137.243283] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2137.596198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2137.599305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2137.638967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2138.682052] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2138.760256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2140.077375] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2140.158653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2142.691931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.692608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.693857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.695265] LustreError: Skipped 1 previous similar message [ 2142.699879] Lustre: Skipped 1 previous similar message [ 2146.863245] LustreError: 123579:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2146.944634] Lustre: server umount lustre-OST0000 complete [ 2149.344136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2149.347190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.351506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.353249] Lustre: Skipped 1 previous similar message [ 2152.928542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.928931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.366549] LustreError: 123781:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2154.368713] LustreError: 123781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2154.400642] Lustre: server umount lustre-MDT0000 complete [ 2155.775698] LustreError: 120665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697575 with bad export cookie 9017683699841624321 [ 2155.778040] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.779926] LustreError: 120665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2155.851386] LustreError: 123983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2155.853570] LustreError: 123983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2155.954690] Lustre: server umount lustre-MDT0001 complete [ 2158.931444] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:46:18 (1755697578) [ 2161.238938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.425894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.770052] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2165.709107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2165.840235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.253980] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2169.158552] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2170.483990] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2175.970213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2175.970311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.971683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.971690] Lustre: Skipped 1 previous similar message [ 2175.981225] Lustre: Skipped 1 previous similar message [ 2177.406150] LustreError: 125985:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2177.407990] LustreError: 125985:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2177.436978] Lustre: server umount lustre-MDT0000 complete [ 2178.761625] LustreError: 124632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697598 with bad export cookie 9017683699841625189 [ 2178.764311] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2178.764534] LustreError: 124632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2178.924357] Lustre: server umount lustre-MDT0001 complete [ 2181.703705] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:46:41 (1755697601) [ 2183.969985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2201.439581] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2207.583461] LustreError: 126830:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 2237.279393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.350977] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2241.277708] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.981783] LustreError: 127488:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2241.984433] LustreError: 127488:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2242.059783] Lustre: server umount lustre-OST0000 complete [ 2244.894024] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:47:44 (1755697664) [ 2247.268204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2247.445772] LustreError: 128158:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.454855] LustreError: 128158:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2248.858483] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2251.782828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2253.292191] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2255.076646] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2256.358228] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2258.651866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2258.774309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.777468] Lustre: Skipped 2 previous similar messages [ 2260.819899] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2262.721316] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2277.346275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.350639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.350920] Lustre: Skipped 2 previous similar messages [ 2277.352401] Lustre: Skipped 4 previous similar messages [ 2281.084190] LustreError: 130405:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2281.085908] LustreError: 130405:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2281.111611] Lustre: server umount lustre-MDT0000 complete [ 2282.407528] LustreError: 128138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697702 with bad export cookie 9017683699841625840 [ 2282.411397] LustreError: 128138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2282.412649] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.464323] LustreError: 130302:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.562932] Lustre: server umount lustre-MDT0001 complete [ 2290.031959] Lustre: server umount lustre-OST0000 complete [ 2292.892818] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:48:32 (1755697712) [ 2295.280994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.456176] LustreError: 131482:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.485770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.931085] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2299.901409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2301.395315] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2303.225961] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2304.539832] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2306.935760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2309.084608] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2310.972829] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2312.294320] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2320.359182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2320.553342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2321.845670] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2321.924166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2325.475988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.476729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.477995] LustreError: Skipped 1 previous similar message [ 2325.478466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.483876] Lustre: Skipped 2 previous similar messages [ 2325.486029] Lustre: Skipped 2 previous similar messages [ 2328.622051] LustreError: 134252:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2328.623938] LustreError: 134252:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2328.701661] Lustre: server umount lustre-OST0000 complete [ 2336.087712] Lustre: server umount lustre-MDT0000 complete [ 2337.351391] LustreError: 133086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697757 with bad export cookie 9017683699841626855 [ 2337.354640] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.355248] LustreError: 133086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2337.509423] Lustre: server umount lustre-MDT0001 complete [ 2340.278120] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:49:19 (1755697759) [ 2342.539417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2359.007471] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2365.151495] LustreError: 135298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 2394.847358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.849722] Lustre: Skipped 2 previous similar messages [ 2396.856369] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2398.706379] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2400.989803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.157265] LustreError: 135325:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.205035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2402.512521] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2405.499308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2407.096579] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2408.897340] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2410.220553] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2411.552236] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2411.623102] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2412.962394] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2413.043563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2414.049776] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.051153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2414.054445] Lustre: Skipped 3 previous similar messages [ 2414.060234] Lustre: Skipped 7 previous similar messages [ 2419.772510] LustreError: 138043:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2419.774071] LustreError: 138043:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2419.797885] Lustre: server umount lustre-MDT0000 complete [ 2421.060372] LustreError: 136245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697841 with bad export cookie 9017683699841627863 [ 2421.063572] LustreError: 136245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2421.063959] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2421.218629] Lustre: server umount lustre-MDT0001 complete [ 2432.762774] Lustre: server umount lustre-OST0000 complete [ 2435.505850] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:50:54 (1755697854) [ 2437.773136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2454.239567] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2460.383506] LustreError: 139090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 2490.969653] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2492.772325] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2495.102399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.257356] LustreError: 139117:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.311235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2496.673123] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2499.620809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2501.048125] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2502.896335] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2504.152507] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2506.611691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2506.648358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2506.709284] Lustre: lustre-OST0001: new disk, initializing [ 2506.711618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2508.547683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2508.551828] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2508.593715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2508.773352] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2511.993653] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2513.424125] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2513.510651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2514.891162] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2514.969503] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2515.935948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2515.937822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.940558] Lustre: Skipped 1 previous similar message [ 2515.943274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2515.945141] Lustre: Skipped 3 previous similar messages [ 2521.709104] LustreError: 142786:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2521.711435] LustreError: 142786:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2521.728269] Lustre: server umount lustre-OST0000 complete [ 2524.129626] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2524.131473] LustreError: 139105:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.132655] LustreError: Skipped 1 previous similar message [ 2524.137948] LustreError: 139105:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2529.139689] Lustre: server umount lustre-OST0001 complete [ 2536.601568] Lustre: server umount lustre-MDT0000 complete [ 2537.943364] LustreError: 140039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697958 with bad export cookie 9017683699841628808 [ 2537.947336] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2537.947730] LustreError: 140039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2538.127244] Lustre: server umount lustre-MDT0001 complete [ 2540.168810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2541.468570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.751305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2544.086682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2546.238883] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:52:45 (1755697965) [ 2546.770623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2547.280241] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2547.837706] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:52:47 (1755697967) [ 2550.087167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.194498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2550.205590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2550.283142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.285762] Lustre: Skipped 6 previous similar messages [ 2551.568583] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2554.455309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2554.485562] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2554.495731] Lustre: 145032:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2555.888581] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2557.598162] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2558.750331] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2561.013563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2561.099825] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2563.076831] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2564.833294] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2564.939352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2566.239510] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2566.330860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2567.642420] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2567.719589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2570.209828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.212515] LustreError: Skipped 2 previous similar messages [ 2574.519487] Lustre: server umount lustre-OST0000 complete [ 2577.885414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2579.969477] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2581.837333] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2583.415112] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2590.308642] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.312754] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.153@tcp (at 0@lo) [ 2590.312769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.669263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2591.992870] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2592.071572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2593.226211] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88c160066000.ost_server_uuid 50 [ 2593.714464] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88c160066000.ost_server_uuid in FULL state after 0 sec [ 2593.767024] LustreError: 145038:0:(ldlm_lockd.c:728:ldlm_handle_ast_error()) ### client (nid 192.168.202.53@tcp) returned error from blocking AST (req@0000000055730d6e x1840982286060416 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000294d933f/0x7d253f8c19e403cd lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.53@tcp remote: 0xf2a11460b4de360d expref: 5 pid: 145039 timeout: 2693 lvb_type: 0 [ 2593.777775] LustreError: lustre-MDT0000: A client on nid 192.168.202.53@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2593.780794] LustreError: 145027:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.53@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000294d933f/0x7d253f8c19e403cd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.53@tcp remote: 0xf2a11460b4de360d expref: 6 pid: 145039 timeout: 0 lvb_type: 0 [ 2595.811211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.811911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.816178] Lustre: Skipped 8 previous similar messages [ 2595.817983] Lustre: Skipped 9 previous similar messages [ 2596.978746] Lustre: server umount lustre-OST0000 complete [ 2600.349130] Lustre: server umount lustre-MDT0000 complete [ 2600.929938] LustreError: 145042:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.935748] LustreError: 145042:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2601.621339] LustreError: 145022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698021 with bad export cookie 9017683699841629984 [ 2601.624799] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.625877] LustreError: 145022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2607.829404] Lustre: server umount lustre-MDT0001 complete [ 2611.532804] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2612.614187] Key type lgssc unregistered [ 2612.751403] LNet: 150490:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.753432] LNetError: 150490:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2612.762328] LNet: Removed LNI 192.168.202.153@tcp [ 2613.095642] Key type .llcrypt unregistered [ 2613.096786] Key type ._llcrypt unregistered [ 2621.770036] Key type ._llcrypt registered [ 2621.771082] Key type .llcrypt registered [ 2621.818012] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2622.246778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2622.294343] alg: No test for adler32 (adler32-zlib) [ 2623.175395] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2623.266250] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2624.855206] Key type lgssc registered [ 2625.262680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2627.912210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2627.918390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.083779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.415505] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2633.297880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2633.420726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.706311] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2636.579553] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2637.751465] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2639.944799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2640.072608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2641.945321] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2643.707358] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2655.465130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2661.558901] Lustre: Failing over lustre-MDT0000 [ 2661.721158] LustreError: 154101:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2661.755485] Lustre: server umount lustre-MDT0000 complete [ 2662.423685] LustreError: 154103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.53@tcp arrived at 1755698082 with bad export cookie 7356177249569079372 [ 2662.428394] LustreError: 154103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2664.415816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2664.418327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.422657] LustreError: 151853:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.426714] LustreError: 151853:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2664.719700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.761302] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2664.856695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.888961] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2666.177204] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2666.933028] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2666.935768] Lustre: lustre-MDT0000: Denying connection for new client 02082398-bcf8-4804-96ab-a2d8775899e4 (at 192.168.202.53@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2670.054900] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 2674.143966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2674.146362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.149486] Lustre: Skipped 2 previous similar messages [ 2674.151304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.168981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.171054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.790051] LustreError: 155007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2679.791982] LustreError: 155007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2679.862975] Lustre: server umount lustre-OST0000 complete [ 2681.107067] LustreError: 155208:0:(ldlm_lib.c:2916:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2681.110620] Lustre: 154581:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2681.113069] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2681.118921] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2681.122446] LustreError: 154581:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003c8e179e x1840982831172736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2681.129341] LustreError: 154581:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2681.132327] LustreError: 154581:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2681.135168] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2681.210157] LustreError: 155208:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2681.211747] LustreError: 155208:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2681.237920] Lustre: server umount lustre-MDT0000 complete [ 2682.471230] LustreError: 155209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698102 with bad export cookie 7356177249569079701 [ 2682.475557] LustreError: 151164:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2682.476026] LustreError: 155209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2682.476421] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2688.572835] LustreError: 155410:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2688.575991] LustreError: 155410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2688.656842] Lustre: server umount lustre-MDT0001 complete [ 2692.134270] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2693.311455] Key type lgssc unregistered [ 2693.454413] LNet: 155937:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2693.457420] LNetError: 155937:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2693.466337] LNet: Removed LNI 192.168.202.153@tcp [ 2693.789972] Key type .llcrypt unregistered [ 2693.790907] Key type ._llcrypt unregistered [ 2702.440761] Key type ._llcrypt registered [ 2702.441887] Key type .llcrypt registered [ 2702.478169] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2702.866374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2702.911105] alg: No test for adler32 (adler32-zlib) [ 2703.766843] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2703.860793] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2705.447189] Key type lgssc registered [ 2705.841746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2708.532936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2708.539621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.708361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.071918] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2713.811643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2713.936270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2715.214947] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2717.009127] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.239621] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2720.459950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2720.588263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.394076] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2724.106895] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2733.926747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2739.170902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2739.170931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2739.174742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.179118] Lustre: Skipped 1 previous similar message [ 2741.867096] LustreError: 159549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2741.930752] Lustre: server umount lustre-OST0000 complete [ 2744.289317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.290179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.292516] Lustre: Skipped 2 previous similar messages [ 2749.247112] LustreError: 159752:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2749.249871] LustreError: 159752:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2749.284862] Lustre: server umount lustre-MDT0000 complete [ 2749.408563] LustreError: 157300:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.412840] LustreError: 157300:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2750.492043] LustreError: 159754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698170 with bad export cookie 8302298754128474474 [ 2750.494740] LustreError: 159754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2750.495385] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.562407] LustreError: 159955:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2750.565030] LustreError: 159955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2750.642577] Lustre: server umount lustre-MDT0001 complete [ 2754.074124] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2755.214317] Key type lgssc unregistered [ 2755.345421] LNet: 160481:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.349468] LNetError: 160481:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.362420] LNet: Removed LNI 192.168.202.153@tcp [ 2755.671585] Key type .llcrypt unregistered [ 2755.673589] Key type ._llcrypt unregistered [ 2767.503137] Key type ._llcrypt registered [ 2767.504042] Key type .llcrypt registered [ 2767.542578] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2767.935460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2767.995706] alg: No test for adler32 (adler32-zlib) [ 2768.883704] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2768.988889] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2770.583136] Key type lgssc registered [ 2771.055470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2773.997923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2774.005910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.169460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.482780] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2779.461932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2779.580222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2780.852502] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2782.663209] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2783.885623] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2786.116541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2786.246188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.068535] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2789.825300] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2800.620581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2809.824449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2809.828123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.835094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.848984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.849277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2810.850105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.850628] Lustre: Skipped 1 previous similar message [ 2812.653394] LustreError: 164381:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2812.718413] Lustre: server umount lustre-OST0000 complete [ 2815.456037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2815.459629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.465912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.088264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2821.089922] Lustre: Skipped 2 previous similar messages [ 2826.208437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.210959] Lustre: Skipped 1 previous similar message [ 2828.255170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2828.289425] LustreError: 164584:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2828.291100] LustreError: 164584:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2828.320276] Lustre: server umount lustre-MDT0000 complete [ 2829.626535] LustreError: 163740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698249 with bad export cookie 2760736408935312706 [ 2829.628879] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2829.629654] LustreError: 163740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2829.707644] LustreError: 164787:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2829.710062] LustreError: 164787:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2829.806627] Lustre: server umount lustre-MDT0001 complete [ 2833.666320] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2834.829340] Key type lgssc unregistered [ 2834.967511] LNet: 165313:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2834.969971] LNetError: 165313:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2834.981546] LNet: Removed LNI 192.168.202.153@tcp [ 2835.362720] Key type .llcrypt unregistered [ 2835.363833] Key type ._llcrypt unregistered [ 2844.233974] Key type ._llcrypt registered [ 2844.234963] Key type .llcrypt registered [ 2844.273154] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2844.609890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2844.620546] alg: No test for adler32 (adler32-zlib) [ 2845.488379] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2845.578897] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2847.167246] Key type lgssc registered [ 2847.647289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2851.412166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2851.422071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.575789] Lustre: *** cfs_fail_loc=135, val=0*** [ 2852.620931] LustreError: 166785:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000 failed (-2) [ 2852.622967] LustreError: 166785:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.202.153@tcp: cfg command failed: rc = -2 [ 2852.625303] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2852.625303] [ 2852.627897] LustreError: MGC192.168.202.153@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2852.631817] LustreError: 166746:0:(tgt_mount.c:1710:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2852.634093] LustreError: 166746:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 2852.636131] LustreError: 166746:0:(obd_config.c:832:class_cleanup()) Device 5 not setup [ 2852.638751] LustreError: 166746:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2852.708868] Lustre: server umount lustre-MDT0000 complete [ 2852.710197] LustreError: 166746:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2855.541591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.776175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.460727] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2861.285054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2861.444584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.870127] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2864.776313] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.866267] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2866.655901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2866.657872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.661234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.753352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.754339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.776854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.779245] Lustre: Skipped 2 previous similar messages [ 2872.586510] LustreError: 168488:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2872.590128] LustreError: 168488:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2872.624487] Lustre: server umount lustre-MDT0000 complete [ 2874.086683] LustreError: 168491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698294 with bad export cookie 11113442067938125637 [ 2874.092626] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2874.096373] LustreError: 168491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2874.197469] LustreError: 168692:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2874.201440] LustreError: 168692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2874.302576] Lustre: server umount lustre-MDT0001 complete [ 2878.619174] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2879.914768] Key type lgssc unregistered [ 2880.058600] LNet: 169219:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2880.062399] LNetError: 169219:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2880.073506] LNet: Removed LNI 192.168.202.153@tcp [ 2880.516102] Key type .llcrypt unregistered [ 2880.517856] Key type ._llcrypt unregistered [ 2888.760946] Key type ._llcrypt registered [ 2888.762817] Key type .llcrypt registered [ 2888.814419] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 2888.957187] Key type .llcrypt unregistered [ 2888.958893] Key type ._llcrypt unregistered [ 2896.867461] Key type ._llcrypt registered [ 2896.868597] Key type .llcrypt registered [ 2896.904083] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 2897.267448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2897.310891] alg: No test for adler32 (adler32-zlib) [ 2898.167695] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2898.258979] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 2899.839174] Key type lgssc registered [ 2900.217574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2902.949170] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2902.956132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2920.479405] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2926.623293] LustreError: 171274:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.153@tcp: failed processing log, type 4: rc = -110 [ 2956.319190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2958.249200] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2960.078690] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.883554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.090168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2963.114976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2964.408069] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2968.082296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2968.273684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.709615] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 2971.795374] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2973.121587] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2974.152082] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2980.838580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.842531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.845220] Lustre: Skipped 1 previous similar message [ 2985.957798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2985.961873] Lustre: Skipped 2 previous similar messages [ 2986.417122] LustreError: 173931:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2986.520518] Lustre: server umount lustre-OST0000 complete [ 2989.024459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2989.029448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.035638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.072302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.073186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.236420] LustreError: 174133:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2994.238795] LustreError: 174133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2994.268090] Lustre: server umount lustre-MDT0000 complete [ 2995.543142] LustreError: 171283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698415 with bad export cookie 3042984937144165648 [ 2995.546245] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2995.547357] LustreError: 171283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2995.614076] LustreError: 174336:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2995.615896] LustreError: 174336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2995.703353] Lustre: server umount lustre-MDT0001 complete [ 2999.284537] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3000.457104] Key type lgssc unregistered [ 3000.603396] LNet: 174862:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3000.606560] LNetError: 174862:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3000.618520] LNet: Removed LNI 192.168.202.153@tcp [ 3000.947121] Key type .llcrypt unregistered [ 3000.948652] Key type ._llcrypt unregistered [ 3010.165667] Key type ._llcrypt registered [ 3010.167419] Key type .llcrypt registered [ 3010.209603] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3010.690255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3010.719584] alg: No test for adler32 (adler32-zlib) [ 3011.588230] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3011.680893] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3013.265377] Key type lgssc registered [ 3013.660581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3016.189435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3016.198820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.353433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3018.644350] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3022.287871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3022.440586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.979309] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3026.139438] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3027.683801] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3030.546269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3030.731893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.618472] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3034.337642] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3042.023046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3045.977764] Lustre: Failing over lustre-MDT0000 [ 3046.042739] LustreError: 178537:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3046.066632] Lustre: server umount lustre-MDT0000 complete [ 3047.392905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3047.394729] LustreError: 176219:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.395801] Lustre: Skipped 2 previous similar messages [ 3047.401095] LustreError: 176219:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3050.040767] LustreError: 176219:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.512296] LustreError: 176224:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.516616] LustreError: 176224:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3055.153693] LustreError: 176219:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.53@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.021905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.060266] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3059.165247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.195271] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3060.273830] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3060.521698] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3062.011518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3064.295854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3064.302305] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3064.323495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3066.580734] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3071.969101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3071.972039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.977259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.015906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3074.018024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.021624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3075.755666] LustreError: 179805:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3075.758317] LustreError: 179805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3075.828958] Lustre: server umount lustre-OST0000 complete [ 3079.649149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.649261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.650024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.650029] Lustre: Skipped 1 previous similar message [ 3079.652305] Lustre: Skipped 1 previous similar message [ 3083.262404] LustreError: 180010:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3083.264721] LustreError: 180010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3083.289471] Lustre: server umount lustre-MDT0000 complete [ 3084.565407] LustreError: 176206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698504 with bad export cookie 17009496412294547954 [ 3084.569324] LustreError: 176206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3084.569442] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.733561] Lustre: server umount lustre-MDT0001 complete [ 3088.666964] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3089.761106] Key type lgssc unregistered [ 3089.905367] LNet: 180739:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3089.907690] LNetError: 180739:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3089.914409] LNet: Removed LNI 192.168.202.153@tcp [ 3090.260072] Key type .llcrypt unregistered [ 3090.260992] Key type ._llcrypt unregistered [ 3099.211240] Key type ._llcrypt registered [ 3099.212262] Key type .llcrypt registered [ 3099.248430] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3099.572448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.606083] alg: No test for adler32 (adler32-zlib) [ 3100.452141] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3100.528506] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3102.111236] Key type lgssc registered [ 3102.481372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3105.045581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3105.051807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.199395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.508305] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3110.375340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3110.489565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3111.774951] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3113.680705] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3115.049736] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3117.451730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3117.584747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.508816] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3121.428299] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3130.915157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3132.561261] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3142.187535] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3150.258417] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3161.572471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.574081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.575661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3161.576122] LustreError: Skipped 1 previous similar message [ 3161.582377] Lustre: Skipped 1 previous similar message [ 3165.613153] LustreError: 184525:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3165.686308] Lustre: server umount lustre-OST0000 complete [ 3171.809059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.809157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3171.809762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.809768] Lustre: Skipped 1 previous similar message [ 3171.812112] Lustre: Skipped 1 previous similar message [ 3172.992371] LustreError: 184727:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3172.994427] LustreError: 184727:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3173.019150] Lustre: server umount lustre-MDT0000 complete [ 3174.285461] LustreError: 182082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698594 with bad export cookie 2142392293816659650 [ 3174.289281] LustreError: 182082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3174.290916] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.360425] LustreError: 184929:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3174.362367] LustreError: 184929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3174.465414] Lustre: server umount lustre-MDT0001 complete [ 3178.306362] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3179.511488] Key type lgssc unregistered [ 3179.665401] LNet: 185455:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3179.667577] LNetError: 185455:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3179.675401] LNet: Removed LNI 192.168.202.153@tcp [ 3180.014030] Key type .llcrypt unregistered [ 3180.015072] Key type ._llcrypt unregistered [ 3189.159102] Key type ._llcrypt registered [ 3189.160144] Key type .llcrypt registered [ 3189.206800] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3189.693752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3189.701227] alg: No test for adler32 (adler32-zlib) [ 3190.559690] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3190.655248] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3192.239216] Key type lgssc registered [ 3192.661704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3195.465374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3195.472784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.651923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.013249] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3200.891639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3201.015921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.327297] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3204.157581] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3205.357152] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3207.627929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3207.768430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.579847] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3211.382834] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3221.094637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3223.390303] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3232.395389] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3241.636974] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3250.547203] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3257.315855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.319612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3257.319909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3257.324767] Lustre: Skipped 1 previous similar message [ 3262.434026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3262.435677] Lustre: Skipped 1 previous similar message [ 3267.555641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.557984] Lustre: Skipped 1 previous similar message [ 3271.135109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3271.158093] LustreError: 191035:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3271.229347] Lustre: server umount lustre-OST0000 complete [ 3272.672609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.672733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3272.673364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.678040] Lustre: Skipped 2 previous similar messages [ 3272.681330] LustreError: Skipped 1 previous similar message [ 3277.792158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.794371] Lustre: Skipped 2 previous similar messages [ 3278.590979] LustreError: 191238:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3278.593033] LustreError: 191238:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3278.624521] Lustre: server umount lustre-MDT0000 complete [ 3279.862264] LustreError: 186797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698700 with bad export cookie 5877955836745702790 [ 3279.864940] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3279.866586] LustreError: 186797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3279.944335] LustreError: 191441:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3279.947580] LustreError: 191441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3280.037310] Lustre: server umount lustre-MDT0001 complete [ 3283.773622] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3284.819717] Key type lgssc unregistered [ 3284.961379] LNet: 191966:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3284.963525] LNetError: 191966:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3284.971330] LNet: Removed LNI 192.168.202.153@tcp [ 3285.279893] Key type .llcrypt unregistered [ 3285.281030] Key type ._llcrypt unregistered [ 3294.431574] Key type ._llcrypt registered [ 3294.432843] Key type .llcrypt registered [ 3294.474455] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3294.916914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3294.933715] alg: No test for adler32 (adler32-zlib) [ 3295.799269] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3295.891349] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3297.471205] Key type lgssc registered [ 3297.825866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3300.353433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3300.359325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.511219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.777042] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3305.529514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3305.652479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3306.903095] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3308.623863] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3309.831448] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3312.100268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3312.240894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.088561] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3315.846197] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3327.590106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3336.159884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3336.162195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3336.166050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.209569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.209813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.210863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3340.973836] LustreError: 195727:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3341.049564] Lustre: server umount lustre-OST0000 complete [ 3343.329933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.334817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.337191] Lustre: Skipped 2 previous similar messages [ 3348.448309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3348.450303] Lustre: Skipped 1 previous similar message [ 3348.484169] LustreError: 195929:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3348.486273] LustreError: 195929:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3348.512423] Lustre: server umount lustre-MDT0000 complete [ 3349.766254] LustreError: 193311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698769 with bad export cookie 15674519513035292273 [ 3349.769147] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3349.769278] LustreError: 193311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3349.835962] LustreError: 196131:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3349.838072] LustreError: 196131:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3349.916396] Lustre: server umount lustre-MDT0001 complete [ 3353.488025] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3354.598827] Key type lgssc unregistered [ 3354.737427] LNet: 196657:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3354.741440] LNetError: 196657:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3354.751392] LNet: Removed LNI 192.168.202.153@tcp [ 3355.065636] Key type .llcrypt unregistered [ 3355.066810] Key type ._llcrypt unregistered [ 3362.629023] Key type ._llcrypt registered [ 3362.630639] Key type .llcrypt registered [ 3362.670332] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3363.085862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3363.102131] alg: No test for adler32 (adler32-zlib) [ 3363.953614] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3364.043748] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3365.623239] Key type lgssc registered [ 3366.062819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3368.700629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3368.707458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3369.865110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.187903] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3374.122209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3374.248127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3375.711662] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3377.662849] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3379.009128] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3381.356718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3381.511776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.597149] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3385.482905] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3394.856072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3400.160644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3400.161390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.164245] LustreError: Skipped 1 previous similar message [ 3400.170911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.173417] Lustre: Skipped 1 previous similar message [ 3401.645105] LustreError: 200272:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3401.724300] Lustre: server umount lustre-OST0000 complete [ 3405.183356] LustreError: 200473:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3405.186568] LustreError: 200473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3405.221206] Lustre: server umount lustre-MDT0000 complete [ 3405.280086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3405.282979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.285599] LustreError: 198581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.288412] Lustre: Skipped 2 previous similar messages [ 3405.299683] LustreError: 198581:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3406.472084] LustreError: 197856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698826 with bad export cookie 2221211080545406658 [ 3406.474402] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.475892] LustreError: 197856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3406.544324] LustreError: 200676:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3406.546277] LustreError: 200676:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3406.633111] Lustre: server umount lustre-MDT0001 complete [ 3410.428663] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3411.582334] Key type lgssc unregistered [ 3411.724473] LNet: 201202:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.726848] LNetError: 201202:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.738347] LNet: Removed LNI 192.168.202.153@tcp [ 3412.080068] Key type .llcrypt unregistered [ 3412.081174] Key type ._llcrypt unregistered [ 3419.585102] Key type ._llcrypt registered [ 3419.586449] Key type .llcrypt registered [ 3419.625690] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3419.965270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3420.027112] alg: No test for adler32 (adler32-zlib) [ 3420.902612] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3420.991347] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3422.575153] Key type lgssc registered [ 3422.972961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3425.638992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3425.647649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.800591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.089476] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3431.010679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3431.154367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3432.641648] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3434.489204] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3435.717583] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3438.050780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3438.186455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.075368] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3441.876166] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3452.522875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3460.257726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3460.283314] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3460.353647] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3462.365038] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3464.302755] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3466.902163] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 3477.266374] Lustre: Permanently deactivating lustre-OST0001 [ 3477.272559] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3487.223774] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3489.352663] Lustre: Permanently reactivating lustre-OST0001 [ 3489.358062] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3489.361710] Lustre: Skipped 2 previous similar messages [ 3493.886905] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.892941] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3493.896640] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3493.900339] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.202.153@tcp (at 0@lo) [ 3493.912509] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3498.978863] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3498.982498] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.986997] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3498.988014] Lustre: Skipped 2 previous similar messages [ 3501.613564] LustreError: 205862:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3501.630392] Lustre: server umount lustre-OST0001 complete [ 3503.072316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.074580] LustreError: Skipped 1 previous similar message [ 3503.075741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.079440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.080957] Lustre: Skipped 1 previous similar message [ 3504.099509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.099594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.099913] LustreError: 204004:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.099926] LustreError: 204004:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3509.217311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.218214] LustreError: 204570:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.219383] Lustre: Skipped 2 previous similar messages [ 3509.224923] LustreError: 204570:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3514.338382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.339059] LustreError: 204570:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.340443] Lustre: Skipped 1 previous similar message [ 3517.407097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3517.425112] LustreError: 206063:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3517.427521] LustreError: 206063:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3517.490805] Lustre: server umount lustre-OST0000 complete [ 3519.457966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.462067] Lustre: Skipped 2 previous similar messages [ 3524.576155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.577952] Lustre: Skipped 3 previous similar messages [ 3533.279091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3533.312070] LustreError: 206265:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3533.313988] LustreError: 206265:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3533.337879] Lustre: server umount lustre-MDT0000 complete [ 3534.585570] LustreError: 204929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698954 with bad export cookie 3517919185800028002 [ 3534.588444] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.588955] LustreError: 204929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3534.758467] Lustre: server umount lustre-MDT0001 complete [ 3538.658340] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3539.736125] Key type lgssc unregistered [ 3539.881364] LNet: 206994:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.885325] LNetError: 206994:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3539.896354] LNet: Removed LNI 192.168.202.153@tcp [ 3540.212246] Key type .llcrypt unregistered [ 3540.213603] Key type ._llcrypt unregistered [ 3551.538725] Key type ._llcrypt registered [ 3551.540182] Key type .llcrypt registered [ 3551.584377] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3558.022059] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3558.434840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3558.477081] alg: No test for adler32 (adler32-zlib) [ 3559.344172] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3559.442116] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3561.031191] Key type lgssc registered [ 3561.406427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3564.491814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3566.766724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3568.850211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3570.974383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3573.274374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3573.295169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3573.300263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.380759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3574.391539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3574.426608] Lustre: lustre-MDT0000: new disk, initializing [ 3574.449808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.456197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3575.794148] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3579.965566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3579.989465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3580.031616] Lustre: 210898:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3580.059430] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3580.062235] Lustre: Skipped 1 previous similar message [ 3580.101453] Lustre: lustre-MDT0001: new disk, initializing [ 3580.139709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3580.152943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3580.156051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3581.508348] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3584.682920] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3585.867475] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3588.100936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3588.126613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3588.230448] Lustre: lustre-OST0000: new disk, initializing [ 3588.232498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3588.266124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.881771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3589.884982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3589.917167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3590.179515] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3593.188971] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3594.503984] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3594.582535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3595.857898] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3595.930738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3598.638082] LustreError: 213808:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3598.706814] Lustre: server umount lustre-OST0000 complete [ 3600.354345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.355070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.357563] Lustre: Skipped 1 previous similar message [ 3605.472256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3605.473938] Lustre: Skipped 2 previous similar messages [ 3610.592661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3610.594351] Lustre: Skipped 1 previous similar message [ 3610.597567] LustreError: 214009:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3610.600231] LustreError: 214009:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3610.626723] Lustre: server umount lustre-MDT0000 complete [ 3611.891386] LustreError: 210891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699032 with bad export cookie 12738995631026876511 [ 3611.894829] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3611.895727] LustreError: 210891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3622.372597] LustreError: 214211:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3622.375689] LustreError: 214211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3622.464244] Lustre: server umount lustre-MDT0001 complete [ 3625.185084] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:10:44 (1755699044) [ 3627.512309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.683304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.949592] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3631.823498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3631.947390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.221263] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3635.035574] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3636.174356] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3638.322774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3640.455745] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3642.499176] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3653.748782] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3658.231239] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3663.703339] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3669.185522] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3684.492038] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3684.497099] Lustre: Skipped 1 previous similar message [ 3700.643075] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3700.645439] Lustre: Skipped 1 previous similar message [ 3740.113914] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3740.116902] Lustre: Skipped 4 previous similar messages [ 3807.822992] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3807.825427] Lustre: Skipped 8 previous similar messages [ 3811.808975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3811.810248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.811503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.814318] Lustre: Skipped 1 previous similar message [ 3815.212116] LustreError: 218355:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3815.213955] LustreError: 218355:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3815.267934] Lustre: server umount lustre-OST0000 complete [ 3816.929073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.937853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3816.941446] Lustre: Skipped 1 previous similar message [ 3821.536156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3821.538288] LustreError: Skipped 1 previous similar message [ 3821.539461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.803053] LustreError: 218561:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3822.806894] LustreError: 218561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3822.850873] Lustre: server umount lustre-MDT0000 complete [ 3824.402918] LustreError: 217082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699244 with bad export cookie 12738995631026877379 [ 3824.406634] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3824.410111] LustreError: 217082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3824.583423] Lustre: server umount lustre-MDT0001 complete [ 3828.686095] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3829.910259] Key type lgssc unregistered [ 3830.055363] LNet: 219287:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3830.057549] LNetError: 219287:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3830.065428] LNet: Removed LNI 192.168.202.153@tcp [ 3830.400511] Key type .llcrypt unregistered [ 3830.401486] Key type ._llcrypt unregistered [ 3839.194696] Key type ._llcrypt registered [ 3839.195668] Key type .llcrypt registered [ 3839.235442] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3839.806097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3839.837285] alg: No test for adler32 (adler32-zlib) [ 3840.704272] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3840.785975] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3842.367223] Key type lgssc registered [ 3842.759258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3845.206908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3845.212816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.363798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.544773] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3850.369089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3850.506795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3851.709871] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3853.380112] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3854.461798] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3856.631840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3856.750050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.551271] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_default_debug -1 all [ 3860.160920] Lustre: DEBUG MARKER: oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3873.247988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.250194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.253755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.273397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3874.275074] Lustre: Skipped 2 previous similar messages [ 3877.803087] LustreError: 223044:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3877.877197] Lustre: server umount lustre-OST0000 complete [ 3879.392479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.393272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3879.395961] Lustre: Skipped 2 previous similar messages [ 3885.178404] LustreError: 223246:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3885.181472] LustreError: 223246:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3885.207094] Lustre: server umount lustre-MDT0000 complete [ 3885.537171] LustreError: 220652:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3885.543950] LustreError: 220652:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3886.360418] LustreError: 220633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699306 with bad export cookie 16123233364571661626 [ 3886.363875] LustreError: MGC192.168.202.153@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3886.364417] LustreError: 220633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3886.425889] LustreError: 223447:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3886.427961] LustreError: 223447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3886.502867] Lustre: server umount lustre-MDT0001 complete [ 3890.166285] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3891.324598] Key type lgssc unregistered [ 3891.471422] LNet: 223973:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3891.473445] LNetError: 223973:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3891.479454] LNet: Removed LNI 192.168.202.153@tcp [ 3891.810994] Key type .llcrypt unregistered [ 3891.812636] Key type ._llcrypt unregistered [ 3898.496797] Key type ._llcrypt registered [ 3898.497765] Key type .llcrypt registered [ 3898.530311] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3898.642373] Key type .llcrypt unregistered [ 3898.644278] Key type ._llcrypt unregistered [ 3906.670652] Key type ._llcrypt registered [ 3906.671888] Key type .llcrypt registered [ 3906.714164] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing set_hostid [ 3913.005815] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing load_modules_local [ 3913.387709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3913.417961] alg: No test for adler32 (adler32-zlib) [ 3914.279587] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3914.364430] LNet: Added LNI 192.168.202.153@tcp [8/256/0/180] [ 3915.943208] Key type lgssc registered [ 3916.351701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3919.773312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3922.487304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3924.514394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3926.622388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3927.155719] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3798 sec ========== 10:15:46 (1755699346) [ 3927.755826] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:15:47 (1755699347) === [ 3929.002398] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:15:48 (1755699348) === [ 3942.091187] Lustre: DEBUG MARKER: oleg253-server.virtnet: executing unload_modules_local [ 3943.174332] Key type lgssc unregistered [ 3943.309280] LNet: 230040:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3943.311414] LNetError: 230040:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3943.324299] LNet: Removed LNI 192.168.202.153@tcp [ 3943.638862] Key type .llcrypt unregistered [ 3943.640096] Key type ._llcrypt unregistered