[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504308853 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003080] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011030] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.022139] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027375] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] *** VALIDATE tmpfs *** [ 0.032152] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037017] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043683] debug: unmapping init [mem 0xffffffffa4059000-0xffffffffa4060fff] [ 0.046243] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050011] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053068] ... fixed-purpose events: 3 [ 0.054011] ... event mask: 000000070000000f [ 0.055284] rcu: Hierarchical SRCU implementation. [ 0.057494] smp: Bringing up secondary CPUs ... [ 0.058541] x86: Booting SMP configuration: [ 0.059023] .... node #0, CPUs: #1 #2 #3 [ 0.062275] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.148674] node 0 deferred pages initialised in 81ms [ 0.151599] devtmpfs: initialized [ 0.152273] x86/mm: Memory block size: 128MB [ 0.155874] gcov: version magic: 0x41383552 [ 0.159248] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161459] pinctrl core: initialized pinctrl subsystem [ 0.162269] [ 0.162922] ************************************************************* [ 0.163026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164021] ** ** [ 0.165021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166021] ** ** [ 0.167021] ** This means that this kernel is built to expose internal ** [ 0.168022] ** IOMMU data structures, which may compromise security on ** [ 0.169021] ** your system. ** [ 0.170019] ** ** [ 0.171020] ** If you see this message and you are not debugging the ** [ 0.172021] ** kernel, report this immediately to your vendor! ** [ 0.173019] ** ** [ 0.174020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175020] ************************************************************* [ 0.176867] NET: Registered protocol family 16 [ 0.178601] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188122] cpuidle: using governor menu [ 0.190727] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193524] PCI: Using configuration type 1 for base access [ 0.195133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204220] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210143] cryptd: max_cpu_qlen set to 1000 [ 0.213384] ACPI: Added _OSI(Module Device) [ 0.214018] ACPI: Added _OSI(Processor Device) [ 0.216012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.221049] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226565] ACPI: Interpreter enabled [ 0.228075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229014] ACPI: Using IOAPIC for interrupt routing [ 0.231107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235471] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245454] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259462] acpiphp: Slot [2] registered [ 0.261115] acpiphp: Slot [5] registered [ 0.263119] acpiphp: Slot [6] registered [ 0.264117] acpiphp: Slot [7] registered [ 0.266129] acpiphp: Slot [8] registered [ 0.267175] acpiphp: Slot [9] registered [ 0.269139] acpiphp: Slot [10] registered [ 0.271123] acpiphp: Slot [3] registered [ 0.272092] acpiphp: Slot [4] registered [ 0.274145] acpiphp: Slot [11] registered [ 0.275107] acpiphp: Slot [12] registered [ 0.277118] acpiphp: Slot [13] registered [ 0.278124] acpiphp: Slot [14] registered [ 0.280091] acpiphp: Slot [15] registered [ 0.281100] acpiphp: Slot [16] registered [ 0.283128] acpiphp: Slot [17] registered [ 0.285102] acpiphp: Slot [18] registered [ 0.287106] acpiphp: Slot [19] registered [ 0.288127] acpiphp: Slot [20] registered [ 0.290100] acpiphp: Slot [21] registered [ 0.291121] acpiphp: Slot [22] registered [ 0.293117] acpiphp: Slot [23] registered [ 0.294105] acpiphp: Slot [24] registered [ 0.295067] acpiphp: Slot [25] registered [ 0.296055] acpiphp: Slot [26] registered [ 0.297076] acpiphp: Slot [27] registered [ 0.299120] acpiphp: Slot [28] registered [ 0.301110] acpiphp: Slot [29] registered [ 0.302109] acpiphp: Slot [30] registered [ 0.304110] acpiphp: Slot [31] registered [ 0.305094] PCI host bridge to bus 0000:00 [ 0.306029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.309023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.313018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.318038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.324142] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327420] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338909] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344101] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.347023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356795] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361159] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.364068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367899] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.388027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402408] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457235] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.500028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.528823] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.570021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582305] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.592019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.600016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.623029] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.633455] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.647160] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.653023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.676023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.688320] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.694022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.713019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724180] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.729461] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.732391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.734728] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.739172] iommu: Default domain type: Passthrough [ 0.741409] SCSI subsystem initialized [ 0.742180] ACPI: bus type USB registered [ 0.744116] usbcore: registered new interface driver usbfs [ 0.746083] usbcore: registered new interface driver hub [ 0.747074] usbcore: registered new device driver usb [ 0.749183] pps_core: LinuxPPS API ver. 1 registered [ 0.751014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754068] PTP clock support registered [ 0.756071] EDAC MC: Ver: 3.0.0 [ 0.758180] PCI: Using ACPI for IRQ routing [ 0.760256] NetLabel: Initializing [ 0.762016] NetLabel: domain hash size = 128 [ 0.764013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.765089] NetLabel: unlabeled traffic allowed by default [ 0.768115] vgaarb: loaded [ 0.770314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.771016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.777912] clocksource: Switched to clocksource kvm-clock [ 0.890442] VFS: Disk quotas dquot_6.6.0 [ 0.892295] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.895390] *** VALIDATE ramfs *** [ 0.897039] *** VALIDATE hugetlbfs *** [ 0.899119] pnp: PnP ACPI init [ 0.901831] pnp: PnP ACPI: found 6 devices [ 0.930435] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.934182] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.936575] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.938706] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.941146] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.943591] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.946345] NET: Registered protocol family 2 [ 0.948848] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.952992] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.956453] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.961663] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.965199] TCP: Hash tables configured (established 65536 bind 65536) [ 0.968557] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.972082] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975184] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978540] NET: Registered protocol family 1 [ 0.981342] RPC: Registered named UNIX socket transport module. [ 0.983894] RPC: Registered udp transport module. [ 0.985721] RPC: Registered tcp transport module. [ 0.987736] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.990912] NET: Registered protocol family 44 [ 0.992491] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.994587] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.996811] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.999784] PCI: CLS 0 bytes, default 64 [ 1.002211] Unpacking initramfs... [ 2.437843] debug: unmapping init [mem 0xffff97213cc54000-0xffff97213ffbffff] [ 2.443418] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.447264] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.451680] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.971784] Initialise system trusted keyrings [ 2.974151] Key type blacklist registered [ 2.979175] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.991765] zbud: loaded [ 2.995750] *** VALIDATE nfs *** [ 2.997096] *** VALIDATE nfs4 *** [ 2.998835] pstore: using deflate compression [ 3.003235] Platform Keyring initialized [ 3.111871] NET: Registered protocol family 38 [ 3.113897] Key type asymmetric registered [ 3.115637] Asymmetric key parser 'x509' registered [ 3.117656] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.121195] io scheduler mq-deadline registered [ 3.123233] io scheduler kyber registered [ 3.125096] io scheduler bfq registered [ 3.127415] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.130375] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.132220] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.135118] ACPI: Power Button [PWRF] [ 3.139931] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.145710] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.158290] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.165763] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.194437] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.224163] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.252193] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.258297] Non-volatile memory driver v1.3 [ 3.260917] Linux agpgart interface v0.103 [ 3.296896] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.300126] vda: detected capacity change from 0 to 74326016 [ 3.316557] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.319688] vdb: detected capacity change from 0 to 1073741824 [ 3.337047] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.340423] vdc: detected capacity change from 0 to 2621440000 [ 3.357212] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.360532] vdd: detected capacity change from 0 to 2621440000 [ 3.376623] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.380222] vde: detected capacity change from 0 to 4294967296 [ 3.397464] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.401170] vdf: detected capacity change from 0 to 4294967296 [ 3.409562] libphy: Fixed MDIO Bus: probed [ 3.421504] usbcore: registered new interface driver usbserial_generic [ 3.424154] usbserial: USB Serial support registered for generic [ 3.426839] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.433422] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.436425] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.440154] mousedev: PS/2 mouse device common for all mice [ 3.443589] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.444846] rtc_cmos 00:05: RTC can wake from S4 [ 3.450884] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.452289] rtc_cmos 00:05: registered as rtc0 [ 3.456584] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.460304] intel_pstate: CPU model not supported [ 3.460397] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.467184] hid: raw HID events driver (C) Jiri Kosina [ 3.469386] usbcore: registered new interface driver usbhid [ 3.471869] usbhid: USB HID core driver [ 3.473778] drop_monitor: Initializing network drop monitor service [ 3.476680] Initializing XFRM netlink socket [ 3.478906] NET: Registered protocol family 10 [ 3.482425] Segment Routing with IPv6 [ 3.484351] NET: Registered protocol family 17 [ 3.486555] mpls_gso: MPLS GSO support [ 3.493135] RAS: Correctable Errors collector initialized. [ 3.495654] AVX version of gcm_enc/dec engaged. [ 3.497566] AES CTR mode by8 optimization enabled [ 3.578951] sched_clock: Marking stable (3578924622, 0)->(4475897606, -896972984) [ 3.582636] registered taskstats version 1 [ 3.585755] Loading compiled-in X.509 certificates [ 3.588182] zswap: loaded using pool lzo/zbud [ 3.615591] Key type big_key registered [ 3.629099] Key type encrypted registered [ 3.630976] ima: No TPM chip found, activating TPM-bypass! [ 3.632571] ima: Allocated hash algorithm: sha1 [ 3.633885] ima: No architecture policies found [ 3.635665] evm: Initialising EVM extended attributes: [ 3.637904] evm: security.selinux [ 3.639418] evm: security.ima [ 3.640737] evm: security.capability [ 3.641690] evm: HMAC attrs: 0x1 [ 3.643769] rtc_cmos 00:05: setting system clock to 2026-06-23 13:06:52 UTC (1782220012) [ 3.651151] debug: unmapping init [mem 0xffffffffa5003000-0xffffffffa51fffff] [ 3.654617] debug: unmapping init [mem 0xffffffffa3d82000-0xffffffffa4058fff] [ 3.663207] Write protecting the kernel read-only data: 28672k [ 3.666405] debug: unmapping init [mem 0xffffffffa2403000-0xffffffffa25fffff] [ 3.669313] debug: unmapping init [mem 0xffffffffa2d14000-0xffffffffa2dfffff] [ 3.706701] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.715513] systemd[1]: Detected virtualization kvm. [ 3.717649] systemd[1]: Detected architecture x86-64. [ 3.719534] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.745431] systemd[1]: No hostname configured. [ 3.747766] systemd[1]: Set hostname to . [ 3.750104] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752564] systemd[1]: Initializing machine ID from random generator. [ 3.811929] random: ln: uninitialized urandom read (6 bytes read) [ 3.884421] random: systemd: uninitialized urandom read (16 bytes read) [ 3.886815] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.894948] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.897725] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.494317] device-mapper: uevent: version 1.0.3 [ 4.496892] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.171603] random: fast init done [ 5.180280] virtio_net virtio0 ens2: renamed from eth0 [ 5.218952] scsi host0: ata_piix [ 5.226786] scsi host1: ata_piix [ 5.228599] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.231130] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.561969] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.161768] random: crng init done [ 10.163592] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.489650] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.586504] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.849122] SELinux: Disabled at runtime. [ 11.909646] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.920264] systemd[1]: Detected virtualization kvm. [ 11.922273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.445535] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.448924] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.453746] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.458464] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.461744] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.469780] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.481490] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target[ 12.523188] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS . [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.915422] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.188413] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.258830] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.328496] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.345925] EDAC sbridge: Ver: 1.1.2 [ 15.085391] Key type dns_resolver registered [ 15.388839] NFS: Registering the id_resolver key type [ 15.391140] Key type id_resolver registered [ 15.392252] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg435-server login: [ 41.468344] libcfs: loading out-of-tree module taints kernel. [ 41.489193] Key type ._llcrypt registered [ 41.490541] Key type .llcrypt registered [ 41.552316] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 50.996494] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 51.754188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.764282] alg: No test for adler32 (adler32-zlib) [ 52.820821] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 53.192109] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 54.823200] Key type lgssc registered [ 56.008655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.900744] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.539591] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 138.207865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.244364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.635837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.681637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.808308] Lustre: lustre-MDT0000: new disk, initializing [ 139.946222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.986862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.107982] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.610232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.693092] Lustre: 6484:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.723526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.727672] Lustre: Skipped 1 previous similar message [ 159.818698] Lustre: lustre-MDT0001: new disk, initializing [ 159.894645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.926533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.945656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.940409] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.291031] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.933402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.235849] Lustre: lustre-OST0000: new disk, initializing [ 180.240787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.249317] Lustre: 8421:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.339589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.405066] hrtimer: interrupt took 4147671 ns [ 187.195400] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.945316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.962932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.033331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.242182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.432672] Lustre: lustre-OST0001: new disk, initializing [ 203.435389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.439757] Lustre: 9493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.547150] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.995470] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.568255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 213.576771] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 213.732161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.745500] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.436660] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.121227] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.723360] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing yml_node [ 251.817436] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 254.926745] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 258.168605] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 260.058527] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 09:11:07 EDT 2026 [ 280.901924] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 282.764884] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.465357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.474078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.504803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.024452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.026438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.053109] Lustre: Skipped 3 previous similar messages [ 304.655899] Lustre: server umount lustre-MDT0000 complete [ 308.708421] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220317 with bad export cookie 13362513305816827477 [ 308.714290] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.725059] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 311.272139] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.287316] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.288277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.295618] Lustre: Skipped 3 previous similar messages [ 311.314702] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 313.313641] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.314648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.351607] Lustre: Skipped 2 previous similar messages [ 315.136530] Lustre: server umount lustre-MDT0001 complete [ 319.250516] Lustre: server umount lustre-OST0000 complete [ 322.971035] Lustre: server umount lustre-OST0001 complete [ 330.012922] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 339.999108] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 378.884112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.226507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.276594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 379.373776] Lustre: lustre-MDT0000: new disk, initializing [ 379.502113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.522763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.912473] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 396.079747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.257495] Lustre: 15048:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.373186] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.378241] Lustre: Skipped 1 previous similar message [ 396.516925] Lustre: lustre-MDT0001: new disk, initializing [ 396.644221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.699797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.707305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 401.407336] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 410.582685] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 415.969222] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 422.513446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.852754] Lustre: lustre-OST0000: new disk, initializing [ 422.860549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 422.871940] Lustre: 17020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 422.985317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.674365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 424.684642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 424.813545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 429.961930] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 440.091707] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 444.848669] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.063992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.185571] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.402597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 455.649238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.651795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.681925] Lustre: Skipped 1 previous similar message [ 455.713820] Lustre: Skipped 1 previous similar message [ 457.730069] Lustre: server umount lustre-OST0000 complete [ 463.328948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 463.354839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.888915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 465.906042] Lustre: Skipped 2 previous similar messages [ 469.178662] LustreError: 18559:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (ffff972185481500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 469.319655] Lustre: server umount lustre-MDT0000 complete [ 471.010960] LustreError: 15061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.040907] LustreError: 15061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 473.379352] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220482 with bad export cookie 13362513305816829010 [ 473.380993] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.398496] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 473.788463] Lustre: server umount lustre-MDT0001 complete [ 477.178495] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:14:44 (1782220484) [ 484.526210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.250137] LustreError: 19284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.335860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.473752] LustreError: 19285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.657355] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 499.745266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.277244] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 515.434767] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.905517] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 531.343974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.717882] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 544.079502] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 570.852514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.855490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.879043] Lustre: Skipped 1 previous similar message [ 570.883552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.889172] Lustre: Skipped 1 previous similar message [ 573.118438] Lustre: server umount lustre-OST0000 complete [ 576.991964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.329441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 591.520492] Lustre: server umount lustre-MDT0000 complete [ 594.983532] LustreError: 19264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220603 with bad export cookie 13362513305816829892 [ 594.988086] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.995208] LustreError: 19264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 595.202933] Lustre: server umount lustre-MDT0001 complete [ 606.489878] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 609.880734] Key type lgssc unregistered [ 610.216559] LNet: 22842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.233793] LNetError: 22842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.250978] LNet: Removed LNI 192.168.204.135@tcp [ 611.183281] Key type .llcrypt unregistered [ 611.189230] Key type ._llcrypt unregistered [ 637.436148] Key type ._llcrypt registered [ 637.441615] Key type .llcrypt registered [ 637.540851] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 638.506943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.775111] alg: No test for adler32 (adler32-zlib) [ 639.979081] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 640.369960] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 642.119211] Key type lgssc registered [ 643.291655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 651.848906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 651.868671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.299707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.232559] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 663.532440] LustreError: 24255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.552535] LustreError: 24255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 667.917618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.390615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.225410] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 680.984565] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 687.638284] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 696.556393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.091086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.143341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 704.557470] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 712.609609] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 730.080739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 730.094966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.111761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 731.346946] Lustre: server umount lustre-OST0000 complete [ 739.302459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 739.308147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 739.313256] Lustre: Skipped 1 previous similar message [ 739.324734] Lustre: Skipped 1 previous similar message [ 741.886376] Lustre: server umount lustre-MDT0000 complete [ 744.419756] LustreError: 24254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.438474] LustreError: 24254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 745.599786] LustreError: 24237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220754 with bad export cookie 4249078546729379346 [ 745.628096] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.300817] Lustre: server umount lustre-MDT0001 complete [ 766.450504] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 768.899518] Key type lgssc unregistered [ 769.252988] LNet: 28232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.258937] LNetError: 28232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.279281] LNet: Removed LNI 192.168.204.135@tcp [ 771.081753] Key type .llcrypt unregistered [ 771.083820] Key type ._llcrypt unregistered [ 794.883299] Key type ._llcrypt registered [ 794.885877] Key type .llcrypt registered [ 795.038345] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 796.030911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.120637] alg: No test for adler32 (adler32-zlib) [ 797.167748] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 797.405836] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 799.111174] Key type lgssc registered [ 800.345752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 808.079663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.095253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.618809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.427128] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 819.173199] LustreError: 29646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.195582] LustreError: 29646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 822.393056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.676695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 828.149922] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 837.134262] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 843.729837] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 857.353401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 857.952668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.121222] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 874.895721] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.481299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 887.790795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.794314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.836777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.825838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.826176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.868753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.849313] Lustre: server umount lustre-OST0000 complete [ 899.552360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 899.557902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.579358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.592851] Lustre: Skipped 1 previous similar message [ 903.634885] Lustre: server umount lustre-MDT0000 complete [ 905.188249] LustreError: 30888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.209834] LustreError: 30888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 907.308830] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782220916 with bad export cookie 6917799192303245845 [ 907.313283] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.325021] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.634920] Lustre: server umount lustre-MDT0001 complete [ 920.481673] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 923.251768] Key type lgssc unregistered [ 923.551130] LNet: 33548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.558696] LNetError: 33548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.590525] LNet: Removed LNI 192.168.204.135@tcp [ 924.406407] Key type .llcrypt unregistered [ 924.410299] Key type ._llcrypt unregistered [ 944.254940] Key type ._llcrypt registered [ 944.257732] Key type .llcrypt registered [ 944.367550] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 945.089635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.219955] alg: No test for adler32 (adler32-zlib) [ 946.257709] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 946.512641] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 948.200574] Key type lgssc registered [ 949.153696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 957.227484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 957.242946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.799398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.354827] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 968.169114] LustreError: 34942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.187566] LustreError: 34942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 972.227192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.548672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.934171] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 987.217948] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 993.034097] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 999.803562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1000.229131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.328546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1006.308755] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1014.044844] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1028.065767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.068178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1028.090854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.185288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1033.203319] Lustre: Skipped 2 previous similar messages [ 1034.038859] Lustre: server umount lustre-OST0000 complete [ 1038.310448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.320317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.346706] Lustre: Skipped 2 previous similar messages [ 1043.426494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1043.437198] Lustre: Skipped 2 previous similar messages [ 1048.545103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1048.553281] Lustre: Skipped 1 previous similar message [ 1052.127153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1052.303092] Lustre: server umount lustre-MDT0000 complete [ 1053.663872] LustreError: 34942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.679041] LustreError: 34942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1056.316298] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221065 with bad export cookie 3669294406560390026 [ 1056.338716] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.752590] Lustre: server umount lustre-MDT0001 complete [ 1068.199928] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1070.822751] Key type lgssc unregistered [ 1071.110641] LNet: 38500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.119779] LNetError: 38500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.131224] LNet: Removed LNI 192.168.204.135@tcp [ 1072.008710] Key type .llcrypt unregistered [ 1072.015818] Key type ._llcrypt unregistered [ 1097.288825] Key type ._llcrypt registered [ 1097.292346] Key type .llcrypt registered [ 1097.424280] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1098.676303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1098.754503] alg: No test for adler32 (adler32-zlib) [ 1099.810960] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1100.024791] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1101.687372] Key type lgssc registered [ 1102.824505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1110.850749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1110.875936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.374357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.384757] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1122.786741] LustreError: 39910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.821170] LustreError: 39910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1124.803041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.188143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1130.161084] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1137.990648] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1143.736849] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1150.654644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.050379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.137374] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1157.258688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1164.283807] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1175.016441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1175.025990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.051225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1175.061958] Lustre: Skipped 1 previous similar message [ 1175.086930] Lustre: Skipped 1 previous similar message [ 1176.668994] Lustre: server umount lustre-OST0000 complete [ 1199.591076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.594371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.611237] Lustre: Skipped 1 previous similar message [ 1199.623356] Lustre: Skipped 1 previous similar message [ 1204.533825] Lustre: server umount lustre-MDT0000 complete [ 1204.705968] LustreError: 39911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.064239] LustreError: 39893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221216 with bad export cookie 15879319957870896508 [ 1208.079600] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.569792] Lustre: server umount lustre-MDT0001 complete [ 1226.341302] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1229.502265] Key type lgssc unregistered [ 1229.835776] LNet: 43616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.843456] LNetError: 43616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.886355] LNet: Removed LNI 192.168.204.135@tcp [ 1231.775396] Key type .llcrypt unregistered [ 1231.780296] Key type ._llcrypt unregistered [ 1256.227617] Key type ._llcrypt registered [ 1256.229804] Key type .llcrypt registered [ 1256.312734] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1257.489475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.550589] alg: No test for adler32 (adler32-zlib) [ 1258.618920] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1258.956223] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1260.767666] Key type lgssc registered [ 1261.917835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.857944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.874194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.416467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.823372] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1282.029576] LustreError: 45030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.051206] LustreError: 45030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1283.917231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1284.310329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1288.636573] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1296.862844] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1302.907117] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1311.326566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1311.728066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.861916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1316.932793] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1324.047453] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1335.267230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.282566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1335.295489] Lustre: Skipped 2 previous similar messages [ 1335.302773] Lustre: Skipped 1 previous similar message [ 1336.784561] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1336.798337] Lustre: Skipped 1 previous similar message [ 1340.188859] Lustre: server umount lustre-MDT0000 complete [ 1340.387057] LustreError: 45030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.401904] LustreError: 45030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1341.863038] LustreError: 45026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.381328] LustreError: 45764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221352 with bad export cookie 6842804871286953960 [ 1343.387200] LustreError: 45764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1343.387444] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.519338] LustreError: 45758:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000083a8f240 x1868794447295744/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1345.504093] LustreError: 45025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.510154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.535773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1348.512540] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1348.521489] Lustre: Skipped 1 previous similar message [ 1349.700291] Lustre: server umount lustre-MDT0001 complete [ 1370.268857] Lustre: server umount lustre-OST0000 complete [ 1385.592137] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1388.588245] Key type lgssc unregistered [ 1388.872749] LNet: 48893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.879536] LNetError: 48893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.899079] LNet: Removed LNI 192.168.204.135@tcp [ 1389.681177] Key type .llcrypt unregistered [ 1389.683117] Key type ._llcrypt unregistered [ 1414.666840] Key type ._llcrypt registered [ 1414.668495] Key type .llcrypt registered [ 1414.761388] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1415.755391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1415.780818] alg: No test for adler32 (adler32-zlib) [ 1416.873761] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1417.076105] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1418.751229] Key type lgssc registered [ 1419.735438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1427.939908] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1427.953415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1443.487461] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1448.608138] LustreError: 50280:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 1475.295247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.445213] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1492.344317] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1660.032693] Lustre: server umount lustre-OST0000 complete [ 1678.447530] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1681.640500] Key type lgssc unregistered [ 1681.889764] LNet: 51957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1681.897033] LNetError: 51957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.907056] LNet: Removed LNI 192.168.204.135@tcp [ 1682.730161] Key type .llcrypt unregistered [ 1682.734808] Key type ._llcrypt unregistered [ 1706.017236] Key type ._llcrypt registered [ 1706.019080] Key type .llcrypt registered [ 1706.137699] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1707.347679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1707.373382] alg: No test for adler32 (adler32-zlib) [ 1708.438428] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1708.792212] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1710.520852] Key type lgssc registered [ 1711.644603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.012642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.031775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.459961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.177194] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1731.042274] LustreError: 53374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.071326] LustreError: 53374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1734.365993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.800027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1739.832316] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1747.511964] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1753.516265] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1760.168273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.435431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.559897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1766.810585] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1774.073896] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1776.102736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.113064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.126749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1779.235654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.105602] Lustre: server umount lustre-OST0000 complete [ 1785.837599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1785.850259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.863139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.865891] Lustre: Skipped 1 previous similar message [ 1789.408992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.414145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.842081] Lustre: server umount lustre-MDT0000 complete [ 1794.533571] LustreError: 53370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1794.549505] LustreError: 53370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1795.509413] LustreError: 53354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221804 with bad export cookie 3552714496430592128 [ 1795.520475] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1795.831753] Lustre: server umount lustre-MDT0001 complete [ 1805.952676] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1808.511549] Key type lgssc unregistered [ 1808.827271] LNet: 56903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.840207] LNetError: 56903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.868281] LNet: Removed LNI 192.168.204.135@tcp [ 1809.760984] Key type .llcrypt unregistered [ 1809.763190] Key type ._llcrypt unregistered [ 1832.503782] Key type ._llcrypt registered [ 1832.505794] Key type .llcrypt registered [ 1832.608649] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1833.486229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.600196] alg: No test for adler32 (adler32-zlib) [ 1834.632279] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1834.905817] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1836.631248] Key type lgssc registered [ 1837.899684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1845.761314] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1845.786042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.408361] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1866.527815] LustreError: 58289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 1893.216077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.353687] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1907.485128] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1914.256856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.732222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.829134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1919.252441] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1919.973724] LustreError: 58314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.985630] LustreError: 58314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1925.095523] LustreError: 58315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.661123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.991423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.910357] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1939.010575] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1944.004269] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1947.618615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.621851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.630515] Lustre: Skipped 1 previous similar message [ 1952.062335] Lustre: server umount lustre-OST0000 complete [ 1973.207261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.218888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.231047] Lustre: Skipped 2 previous similar messages [ 1978.337180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1978.341515] Lustre: Skipped 1 previous similar message [ 1979.515605] Lustre: server umount lustre-MDT0000 complete [ 1982.860824] LustreError: 58297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782221991 with bad export cookie 11266061631054636258 [ 1982.879499] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.281650] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1997.670510] Lustre: server umount lustre-MDT0001 complete [ 2009.242592] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2012.209196] Key type lgssc unregistered [ 2012.531569] LNet: 62034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.541597] LNetError: 62034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.565729] LNet: Removed LNI 192.168.204.135@tcp [ 2013.465125] Key type .llcrypt unregistered [ 2013.467509] Key type ._llcrypt unregistered [ 2038.967223] Key type ._llcrypt registered [ 2038.970424] Key type .llcrypt registered [ 2039.059424] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2039.890593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.967174] alg: No test for adler32 (adler32-zlib) [ 2041.019295] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2041.265482] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2042.977987] Key type lgssc registered [ 2043.986254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2050.943651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2050.959941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.363148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.548996] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2062.818128] LustreError: 63447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.834277] LustreError: 63447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2063.963649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.194883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.545278] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2073.506446] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2078.409836] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2084.664380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.042574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.912552] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2097.340458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2098.845803] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2118.112283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2118.121356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.125480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2118.143835] Lustre: Skipped 1 previous similar message [ 2118.160810] Lustre: Skipped 1 previous similar message [ 2123.257243] LustreError: 65577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.275131] LustreError: 65577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2123.336778] Lustre: server umount lustre-OST0000 complete [ 2128.353493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2128.363244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.375275] Lustre: Skipped 1 previous similar message [ 2128.383513] Lustre: Skipped 1 previous similar message [ 2133.475862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2133.484583] Lustre: Skipped 1 previous similar message [ 2138.594394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.608782] Lustre: Skipped 1 previous similar message [ 2141.151160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2141.315887] Lustre: server umount lustre-MDT0000 complete [ 2143.714985] LustreError: 65939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.733099] LustreError: 65939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2144.871316] LustreError: 63427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222153 with bad export cookie 7570134056451568318 [ 2144.875443] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.888231] LustreError: 63427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2145.284168] Lustre: server umount lustre-MDT0001 complete [ 2156.820320] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2159.856975] Key type lgssc unregistered [ 2160.173674] LNet: 67004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.184367] LNetError: 67004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.193590] LNet: Removed LNI 192.168.204.135@tcp [ 2162.204873] Key type .llcrypt unregistered [ 2162.210060] Key type ._llcrypt unregistered [ 2185.921239] Key type ._llcrypt registered [ 2185.924198] Key type .llcrypt registered [ 2186.032844] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2186.999283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.047507] alg: No test for adler32 (adler32-zlib) [ 2188.178332] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2188.432955] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2190.127195] Key type lgssc registered [ 2191.375372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2200.136159] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:43:27 (1782222207) [ 2206.570605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.599645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.867627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.735628] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2217.447133] LustreError: 68687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.456920] LustreError: 68687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2221.316328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.683939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2227.484244] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2235.029478] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2240.284892] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2246.890635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2247.212396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.345066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2253.491394] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2260.754651] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2269.376139] Lustre: Failing over lustre-MDT0000 [ 2269.657780] Lustre: server umount lustre-MDT0000 complete [ 2271.716168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.718603] LustreError: 68683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.726257] Lustre: Skipped 1 previous similar message [ 2271.751983] LustreError: 68683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2276.834698] LustreError: 68688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.847681] LustreError: 68688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2279.266826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.510987] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.735514] Lustre: *** cfs_fail_loc=135, val=0*** [ 2279.833943] LustreError: 71774:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2279.838923] LustreError: 71774:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -2 [ 2279.851371] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2279.851371] [ 2279.869428] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2279.889730] LustreError: 71763:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2279.899501] LustreError: 71763:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2279.907526] LustreError: 71763:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2279.913771] LustreError: 71763:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2279.978278] Lustre: server umount lustre-MDT0000 complete [ 2279.979967] LustreError: 71763:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2281.953884] LustreError: 68688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.979481] LustreError: 68688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2286.407758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.552732] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.794201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.864263] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2288.034757] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2290.727142] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2292.205768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2292.226246] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2292.258832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2297.316652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.328738] Lustre: Skipped 1 previous similar message [ 2297.334029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2297.334818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2297.346792] Lustre: Skipped 1 previous similar message [ 2299.271444] Lustre: server umount lustre-OST0000 complete [ 2307.554530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.559423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.562978] Lustre: Skipped 2 previous similar messages [ 2307.563452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.587420] Lustre: Skipped 1 previous similar message [ 2308.947619] Lustre: server umount lustre-MDT0000 complete [ 2312.588394] LustreError: 71267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222321 with bad export cookie 17033225815087215935 [ 2312.603131] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.607220] LustreError: 71267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2313.013992] Lustre: server umount lustre-MDT0001 complete [ 2323.724783] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2327.256025] Key type lgssc unregistered [ 2327.619132] LNet: 73555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.627678] LNetError: 73555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.643324] LNet: Removed LNI 192.168.204.135@tcp [ 2328.699547] Key type .llcrypt unregistered [ 2328.704406] Key type ._llcrypt unregistered [ 2351.519214] Key type ._llcrypt registered [ 2351.522078] Key type .llcrypt registered [ 2351.616594] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2352.183230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.224578] alg: No test for adler32 (adler32-zlib) [ 2353.207721] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2353.403756] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2355.071419] Key type lgssc registered [ 2356.038564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2365.294312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2365.321871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.052517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.084879] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2376.163282] LustreError: 74971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.180497] LustreError: 74971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2381.281627] LustreError: 74972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.056775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.347435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2386.444837] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2394.332632] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2400.035905] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2407.491314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.976642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.115402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2414.773688] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2421.902923] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2429.019485] Lustre: Failing over lustre-MDT0000 [ 2429.283868] Lustre: server umount lustre-MDT0000 complete [ 2429.900103] LustreError: 74967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2432.486309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.501860] LustreError: 76406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2432.506443] Lustre: Skipped 2 previous similar messages [ 2432.532810] LustreError: 76406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2437.624352] LustreError: 75699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.660458] LustreError: 75699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2440.624553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.813173] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.044242] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2441.270943] LustreError: 78062:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2441.286366] LustreError: 78062:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -9 [ 2441.306868] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2441.306868] [ 2441.314113] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2441.330705] LustreError: 78050:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2441.358749] LustreError: 78050:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2441.389268] LustreError: 78050:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2441.393470] LustreError: 78050:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2441.423212] Lustre: server umount lustre-MDT0000 complete [ 2441.430410] LustreError: 78050:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2447.851013] LustreError: 74972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.869343] LustreError: 74972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2448.631039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.734565] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.905776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.937244] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2450.337360] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2452.680524] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2454.001235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2454.027392] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2454.060793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2463.201670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2463.208299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.232277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.229069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.235343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.490979] Lustre: server umount lustre-OST0000 complete [ 2474.465628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2474.468692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.485190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.485199] Lustre: Skipped 1 previous similar message [ 2474.518070] Lustre: Skipped 1 previous similar message [ 2479.584293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.597855] Lustre: Skipped 2 previous similar messages [ 2484.191199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2484.416733] Lustre: server umount lustre-MDT0000 complete [ 2484.708459] LustreError: 75699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.735706] LustreError: 75699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2488.431730] LustreError: 74952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222497 with bad export cookie 17515186743895803385 [ 2488.434524] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.444232] LustreError: 74952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2488.836397] Lustre: server umount lustre-MDT0001 complete [ 2500.533774] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2502.981583] Key type lgssc unregistered [ 2503.229565] LNet: 79841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.244805] LNetError: 79841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.260970] LNet: Removed LNI 192.168.204.135@tcp [ 2503.867626] Key type .llcrypt unregistered [ 2503.869950] Key type ._llcrypt unregistered [ 2519.234109] Key type ._llcrypt registered [ 2519.236849] Key type .llcrypt registered [ 2519.381675] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2519.730650] Key type .llcrypt unregistered [ 2519.736900] Key type ._llcrypt unregistered [ 2545.168389] Key type ._llcrypt registered [ 2545.171576] Key type .llcrypt registered [ 2545.291661] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2546.464172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.494404] alg: No test for adler32 (adler32-zlib) [ 2547.608372] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2547.899974] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2549.575445] Key type lgssc registered [ 2550.585820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2556.871867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2556.891133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.324880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.897795] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2568.674461] LustreError: 81742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.688719] LustreError: 81742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2572.160794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.469193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2577.694683] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2585.981914] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2591.589994] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2598.517984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.895646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.017076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2605.277047] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2614.148064] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2627.552082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2627.557555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.560392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2627.570948] Lustre: Skipped 1 previous similar message [ 2632.676430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.685124] Lustre: Skipped 1 previous similar message [ 2633.381079] Lustre: server umount lustre-OST0000 complete [ 2637.796885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.799763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.807231] Lustre: Skipped 1 previous similar message [ 2637.809776] Lustre: Skipped 1 previous similar message [ 2642.922617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.945987] Lustre: Skipped 2 previous similar messages [ 2643.282583] Lustre: server umount lustre-MDT0000 complete [ 2647.357990] LustreError: 81724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222656 with bad export cookie 2039836329781243735 [ 2647.357990] LustreError: 82478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222656 with bad export cookie 2039836329781243735 [ 2647.379874] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.401888] LustreError: 81724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2647.959502] Lustre: server umount lustre-MDT0001 complete [ 2660.643722] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2663.661527] Key type lgssc unregistered [ 2664.050501] LNet: 85294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.058959] LNetError: 85294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2664.083777] LNet: Removed LNI 192.168.204.135@tcp [ 2665.067269] Key type .llcrypt unregistered [ 2665.071128] Key type ._llcrypt unregistered [ 2696.139717] Key type ._llcrypt registered [ 2696.146544] Key type .llcrypt registered [ 2696.493548] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2699.035578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2699.409335] alg: No test for adler32 (adler32-zlib) [ 2700.768964] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2701.172139] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2702.855178] Key type lgssc registered [ 2703.907863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2712.128310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2712.157231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.714708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.106967] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2723.810563] LustreError: 86710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.842336] LustreError: 86710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2728.435345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.817914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2733.971329] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2741.970933] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2747.745336] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2755.439341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2755.871449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.987175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2762.558518] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2770.932181] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2784.737204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.740724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2784.752233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2784.768813] Lustre: Skipped 1 previous similar message [ 2787.694692] Lustre: server umount lustre-OST0000 complete [ 2794.982447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.985305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2795.015950] Lustre: Skipped 2 previous similar messages [ 2795.030194] Lustre: Skipped 1 previous similar message [ 2797.713703] Lustre: server umount lustre-MDT0000 complete [ 2800.111154] LustreError: 87917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.133461] LustreError: 87917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2801.659269] LustreError: 86691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222810 with bad export cookie 17356678992450311820 [ 2801.666927] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.684856] LustreError: 86691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2802.006732] Lustre: server umount lustre-MDT0001 complete [ 2813.325681] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2816.043857] Key type lgssc unregistered [ 2816.323516] LNet: 90244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.335738] LNetError: 90244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2816.353460] LNet: Removed LNI 192.168.204.135@tcp [ 2817.278515] Key type .llcrypt unregistered [ 2817.280592] Key type ._llcrypt unregistered [ 2841.812191] Key type ._llcrypt registered [ 2841.816377] Key type .llcrypt registered [ 2841.960659] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2843.427029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2843.441240] alg: No test for adler32 (adler32-zlib) [ 2844.641772] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2844.904905] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2846.607871] Key type lgssc registered [ 2847.727535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2855.434071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2855.474522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.056201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.727955] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2867.185379] LustreError: 91659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.206369] LustreError: 91659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2870.237381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.522416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2874.201374] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2881.504668] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2887.749331] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2894.173413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.569308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2901.330496] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2902.781232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2909.568620] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2922.986929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2922.997590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2923.014836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2928.551282] Lustre: server umount lustre-OST0000 complete [ 2937.313212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2937.329963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2937.358040] Lustre: Skipped 1 previous similar message [ 2937.361768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2937.365516] Lustre: Skipped 2 previous similar messages [ 2938.851166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.853541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2938.865533] Lustre: Skipped 1 previous similar message [ 2940.276818] Lustre: server umount lustre-MDT0000 complete [ 2943.381486] LustreError: 91640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782222952 with bad export cookie 15010931722432808933 [ 2943.382884] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.402811] LustreError: 91640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2943.600787] Lustre: server umount lustre-MDT0001 complete [ 2956.811866] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2960.073180] Key type lgssc unregistered [ 2960.301463] LNet: 95214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2960.318487] LNetError: 95214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2960.334842] LNet: Removed LNI 192.168.204.135@tcp [ 2961.403443] Key type .llcrypt unregistered [ 2961.414706] Key type ._llcrypt unregistered [ 2987.546572] Key type ._llcrypt registered [ 2987.548809] Key type .llcrypt registered [ 2987.642345] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2989.277892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2989.333596] alg: No test for adler32 (adler32-zlib) [ 2990.541696] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2990.900483] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2992.616676] Key type lgssc registered [ 2993.830410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3001.556279] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3001.566921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3017.055448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3022.177262] LustreError: 96603:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 3048.865437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.737087] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3061.440285] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3066.201193] Lustre: server umount lustre-OST0000 complete [ 3074.380355] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:58:01 (1782223081) [ 3082.418047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.877608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.989593] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3088.354045] LustreError: 98241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.373594] LustreError: 98241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3092.450147] LustreError: 98242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.284056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.693243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.416525] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3107.764886] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3112.978839] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3119.360838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.788369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.076841] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3127.988278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3132.913629] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3143.648405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3143.657802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.663046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3143.679641] Lustre: Skipped 1 previous similar message [ 3143.695339] Lustre: Skipped 1 previous similar message [ 3148.768500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.779111] Lustre: Skipped 1 previous similar message [ 3149.384529] Lustre: server umount lustre-OST0000 complete [ 3153.889678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.900717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.903741] Lustre: Skipped 1 previous similar message [ 3153.921953] Lustre: Skipped 1 previous similar message [ 3158.904381] Lustre: server umount lustre-MDT0000 complete [ 3159.019950] LustreError: 98242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.041594] LustreError: 98242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3162.665533] LustreError: 98221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223171 with bad export cookie 9071098252141925966 [ 3162.677609] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.051816] Lustre: server umount lustre-MDT0001 complete [ 3174.489928] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3177.444249] Key type lgssc unregistered [ 3177.755160] LNet: 101931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.764788] LNetError: 101931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.783803] LNet: Removed LNI 192.168.204.135@tcp [ 3178.601655] Key type .llcrypt unregistered [ 3178.603926] Key type ._llcrypt unregistered [ 3216.903123] Key type ._llcrypt registered [ 3216.905899] Key type .llcrypt registered [ 3217.043109] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3218.385780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3218.432437] alg: No test for adler32 (adler32-zlib) [ 3219.550654] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 3219.878697] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3221.631186] Key type lgssc registered [ 3222.876569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3230.429234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3230.461149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.003513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.125514] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3241.442913] LustreError: 104116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.469061] LustreError: 104116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3244.047272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.284282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.888884] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3258.009927] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3265.168503] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3273.333180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.701924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.802837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3279.670611] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3285.940471] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3299.814125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3299.830228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.845462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.343209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.368716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.368783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.386480] Lustre: Skipped 1 previous similar message [ 3304.399659] Lustre: server umount lustre-OST0000 complete [ 3310.563297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.566549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.577181] Lustre: Skipped 1 previous similar message [ 3314.481352] Lustre: server umount lustre-MDT0000 complete [ 3315.680844] LustreError: 105130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.692712] LustreError: 105130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3318.107423] LustreError: 104097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223326 with bad export cookie 5605249311181926897 [ 3318.128399] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.552346] Lustre: server umount lustre-MDT0001 complete [ 3330.690911] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3333.612280] Key type lgssc unregistered [ 3333.980079] LNet: 107683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3333.992465] LNetError: 107683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3334.013075] LNet: Removed LNI 192.168.204.135@tcp [ 3334.816462] Key type .llcrypt unregistered [ 3334.820705] Key type ._llcrypt unregistered [ 3352.434261] Key type ._llcrypt registered [ 3352.438236] Key type .llcrypt registered [ 3352.565387] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3353.545265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3353.609847] alg: No test for adler32 (adler32-zlib) [ 3354.677169] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 3355.009322] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3356.687528] Key type lgssc registered [ 3358.018719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3366.638258] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3366.668523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3382.367729] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3387.487390] LustreError: 108926:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 3414.175298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.795209] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3426.718290] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3434.846892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.346887] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3435.363846] LustreError: 109949:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3435.376302] LustreError: 109949:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3435.386474] LustreError: 109949:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3435.518308] Lustre: server umount lustre-MDT0000 complete [ 3435.521933] LustreError: 109949:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3454.033162] Lustre: server umount lustre-OST0000 complete [ 3463.519697] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3473.531369] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3510.097697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.299197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3510.330944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3510.418969] Lustre: lustre-MDT0000: new disk, initializing [ 3510.500083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.523815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3514.843659] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3525.666157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.827804] Lustre: 112744:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3525.951098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3525.958112] Lustre: Skipped 1 previous similar message [ 3526.126248] Lustre: lustre-MDT0001: new disk, initializing [ 3526.274658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3526.344343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3526.358181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3531.278600] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3545.402536] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3553.473260] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3562.061172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3562.424492] Lustre: lustre-OST0000: new disk, initializing [ 3562.434447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3562.448057] Lustre: 114713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3562.575810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.326992] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3571.294212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3571.305416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3571.414036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3580.319479] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3585.699497] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3585.938537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3590.831343] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3591.054653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3597.282885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3597.289171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.295765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3597.302153] Lustre: Skipped 1 previous similar message [ 3597.325782] Lustre: Skipped 1 previous similar message [ 3599.636234] Lustre: server umount lustre-OST0000 complete [ 3607.531432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.537897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.565931] Lustre: Skipped 1 previous similar message [ 3607.575359] Lustre: Skipped 1 previous similar message [ 3609.993903] Lustre: server umount lustre-MDT0000 complete [ 3614.066842] LustreError: 112734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223622 with bad export cookie 4149248322100709184 [ 3614.083929] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3614.085022] LustreError: 112734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3614.408679] Lustre: server umount lustre-MDT0001 complete [ 3623.744652] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:07:11 (1782223631) [ 3627.146652] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3645.833786] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3657.221671] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3698.634508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.927699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3698.965390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3699.094096] Lustre: lustre-MDT0000: new disk, initializing [ 3699.200863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.232437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3703.822808] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3714.131330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.279699] Lustre: 119615:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3714.355316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3714.358455] Lustre: Skipped 1 previous similar message [ 3714.455830] Lustre: lustre-MDT0001: new disk, initializing [ 3714.584766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3714.633486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3714.645572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3718.728067] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3729.270659] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3736.006688] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3742.683203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.016210] Lustre: lustre-OST0000: new disk, initializing [ 3743.018931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3743.024795] Lustre: 121583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3743.120905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.830280] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3750.455763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3750.464185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3750.557518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3758.684216] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3763.731610] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3764.002749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3768.970584] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3769.230936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3771.368241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3771.371696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.379047] LustreError: Skipped 1 previous similar message [ 3771.380094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3771.414126] Lustre: Skipped 1 previous similar message [ 3776.481803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.502518] Lustre: Skipped 1 previous similar message [ 3777.673547] Lustre: server umount lustre-OST0000 complete [ 3786.214992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3786.230795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.245904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.251886] Lustre: Skipped 1 previous similar message [ 3787.942390] Lustre: server umount lustre-MDT0000 complete [ 3791.842938] LustreError: 119621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.862644] LustreError: 119621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3792.107349] LustreError: 121593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223800 with bad export cookie 4149248322100710066 [ 3792.122853] LustreError: 121593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3792.124494] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.354813] Lustre: server umount lustre-MDT0001 complete [ 3802.076870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3802.488567] LustreError: 123800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.594442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.546063] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3807.720690] LustreError: 123801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.834469] LustreError: 123800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.631754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.225799] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3829.530639] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3835.066385] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3842.603767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3850.291465] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3858.378518] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3870.083156] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3877.876771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.877173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3877.892423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.892438] Lustre: Skipped 1 previous similar message [ 3877.906255] Lustre: Skipped 2 previous similar messages [ 3881.387703] Lustre: server umount lustre-OST0000 complete [ 3888.103438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.124623] Lustre: Skipped 1 previous similar message [ 3892.830750] Lustre: server umount lustre-MDT0000 complete [ 3893.216745] LustreError: 123800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.138384] LustreError: 126475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223905 with bad export cookie 4149248322100710948 [ 3897.145225] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3897.149804] LustreError: 126475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3897.579043] Lustre: server umount lustre-MDT0001 complete [ 3911.533636] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3914.850551] Key type lgssc unregistered [ 3915.193405] LNet: 127429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3915.209372] LNetError: 127429:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3915.237950] LNet: Removed LNI 192.168.204.135@tcp [ 3916.453529] Key type .llcrypt unregistered [ 3916.455847] Key type ._llcrypt unregistered [ 3938.009469] Key type ._llcrypt registered [ 3938.015638] Key type .llcrypt registered [ 3938.166460] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3956.278550] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3957.394183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3957.570424] alg: No test for adler32 (adler32-zlib) [ 3958.645193] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 3958.916904] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3960.639168] Key type lgssc registered [ 3961.908563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4004.055867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4004.091759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.459213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4005.500794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4005.616671] Lustre: lustre-MDT0000: new disk, initializing [ 4005.719988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4005.739973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4010.747683] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4023.044462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.155419] Lustre: 130587:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4023.199923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4023.203676] Lustre: Skipped 1 previous similar message [ 4023.312868] Lustre: lustre-MDT0001: new disk, initializing [ 4023.430372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4023.476052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4023.507870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4028.906461] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4039.832975] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4047.522511] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4054.755775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4055.113251] Lustre: lustre-OST0000: new disk, initializing [ 4055.125397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4055.138479] Lustre: 132555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4055.297657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.327046] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4063.297050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4063.308678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4063.458877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4072.089106] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.096748] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4078.382690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4084.009752] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4084.402906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4089.313112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.313667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.322390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.339052] Lustre: Skipped 1 previous similar message [ 4089.373858] LustreError: Skipped 1 previous similar message [ 4092.834767] Lustre: server umount lustre-OST0000 complete [ 4099.553939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.573984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.574795] Lustre: Skipped 1 previous similar message [ 4099.591172] Lustre: Skipped 2 previous similar messages [ 4104.672517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.686408] Lustre: Skipped 1 previous similar message [ 4109.795271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.804759] Lustre: Skipped 1 previous similar message [ 4111.327128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4111.614710] Lustre: server umount lustre-MDT0000 complete [ 4115.648357] LustreError: 130579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224124 with bad export cookie 4129919592853952873 [ 4115.652142] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.659706] LustreError: 130579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4115.992824] Lustre: server umount lustre-MDT0001 complete [ 4125.879880] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:15:32 (1782224132) [ 4133.360519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.080422] LustreError: 134969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.098643] LustreError: 134969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4134.189349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.936589] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4139.498615] LustreError: 134970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.586842] LustreError: 134969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.472248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.714078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4148.868706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4154.830694] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4164.905042] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4171.709802] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4174.305367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4174.311187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.324662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4178.913602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.639939] Lustre: server umount lustre-MDT0000 complete [ 4184.033556] LustreError: 134969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.068472] LustreError: 134969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4185.055280] LustreError: 134949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224193 with bad export cookie 4129919592853953755 [ 4185.068886] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.634374] Lustre: server umount lustre-MDT0001 complete [ 4195.685423] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:16:42 (1782224202) [ 4204.192556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4219.679770] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4224.801792] LustreError: 137385:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4250.527347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4257.336425] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4265.641673] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4268.264877] Lustre: server umount lustre-OST0000 complete [ 4277.830245] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:18:04 (1782224284) [ 4286.508805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.175372] LustreError: 138833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.277944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.378543] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4297.697064] LustreError: 138834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.715201] LustreError: 138834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4302.042751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.794907] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4316.329625] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4322.930851] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4330.230723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4330.592815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.609690] Lustre: Skipped 1 previous similar message [ 4337.810513] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4346.445155] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4363.748657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.754581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.756550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4363.767985] Lustre: Skipped 2 previous similar messages [ 4363.792280] Lustre: Skipped 4 previous similar messages [ 4366.823692] Lustre: server umount lustre-MDT0000 complete [ 4368.866949] LustreError: 138833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.887481] LustreError: 138833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4371.283187] LustreError: 138813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224380 with bad export cookie 4129919592853954420 [ 4371.285393] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4371.294722] LustreError: 138813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4371.563563] Lustre: server umount lustre-MDT0001 complete [ 4384.256224] Lustre: server umount lustre-OST0000 complete [ 4394.842337] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:20:01 (1782224401) [ 4402.057085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4402.561760] LustreError: 142507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.649201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.861915] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4416.680988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.566149] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4430.667920] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4437.534455] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.544328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.997569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4455.108393] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4464.352995] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4469.238730] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4469.484403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4474.271908] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4474.539500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4476.898772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4476.906220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.909058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4476.929250] Lustre: Skipped 1 previous similar message [ 4476.942685] Lustre: Skipped 1 previous similar message [ 4491.231234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4491.363412] Lustre: server umount lustre-OST0000 complete [ 4497.381759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.407042] Lustre: Skipped 1 previous similar message [ 4509.151292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4509.568738] Lustre: server umount lustre-MDT0000 complete [ 4512.740127] LustreError: 142504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.764152] LustreError: 142504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4513.985981] LustreError: 142488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224522 with bad export cookie 4129919592853955491 [ 4513.991271] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4514.010688] LustreError: 142488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4514.654693] Lustre: server umount lustre-MDT0001 complete [ 4525.859751] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:22:12 (1782224532) [ 4534.478408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4550.112363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4555.232350] LustreError: 146669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4580.960284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4580.975829] Lustre: Skipped 2 previous similar messages [ 4587.765074] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4595.383613] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4603.002435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.034203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4609.911277] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4620.221530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.561966] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4632.446154] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4638.626541] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4643.619150] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4643.906765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4648.910078] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4649.205743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4651.489231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.491163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4651.507761] Lustre: Skipped 1 previous similar message [ 4651.512823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4651.519741] Lustre: Skipped 12 previous similar messages [ 4657.550646] Lustre: server umount lustre-MDT0000 complete [ 4661.729381] LustreError: 147706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.756135] LustreError: 147706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4661.912212] LustreError: 146677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224670 with bad export cookie 4129919592853956513 [ 4661.920401] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4661.926364] LustreError: 146677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4662.274885] Lustre: server umount lustre-MDT0001 complete [ 4676.775268] Lustre: server umount lustre-OST0000 complete [ 4686.071797] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:24:53 (1782224693) [ 4693.883236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4709.473592] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4714.595347] LustreError: 150866:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4747.118430] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4755.100468] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4761.559245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.151202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4767.004549] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4775.702964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.550587] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4788.694266] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4794.851583] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4801.541297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.673253] Lustre: lustre-OST0001: new disk, initializing [ 4801.676426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4801.680584] Lustre: 153739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4807.361190] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4810.863738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4810.876464] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4811.028215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4820.810584] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4829.015491] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4829.223952] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4833.970770] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4834.202251] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4836.843709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.874384] Lustre: Skipped 1 previous similar message [ 4836.882775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.893278] Lustre: Skipped 4 previous similar messages [ 4842.607428] Lustre: server umount lustre-OST0000 complete [ 4847.072799] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4847.088151] LustreError: Skipped 1 previous similar message [ 4852.751246] Lustre: server umount lustre-OST0001 complete [ 4863.671713] Lustre: server umount lustre-MDT0000 complete [ 4868.729927] LustreError: 150873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224877 with bad export cookie 4129919592853957465 [ 4868.740121] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4868.744144] LustreError: 150873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4869.346675] Lustre: server umount lustre-MDT0001 complete [ 4894.926885] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:28:22 (1782224902) [ 4896.879502] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4898.900742] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4900.730815] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:28:27 (1782224907) [ 4907.561921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.815819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4907.849815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4908.041821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.055665] Lustre: Skipped 6 previous similar messages [ 4913.156846] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4923.477564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.654276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4923.726625] Lustre: 157301:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4929.493214] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4937.296599] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4943.001472] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4950.552221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4950.786548] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4957.773048] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4959.936201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4966.359168] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4971.448324] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4971.797243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4977.199592] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4977.445416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4980.197550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4980.211161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.241757] Lustre: Skipped 5 previous similar messages [ 4985.631241] Lustre: server umount lustre-OST0000 complete [ 5008.717435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.902919] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5020.392929] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.405080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.405522] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5024.786178] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5030.862200] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5031.063755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5036.245473] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5036.459247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5043.368982] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff95bb90201800.ost_server_uuid 50 [ 5045.376378] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff95bb90201800.ost_server_uuid in FULL state after 0 sec [ 5045.549208] LustreError: 157308:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.35@tcp) returned error from blocking AST (req@000000007a8dcb2b x1868797278692736 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000003ca71d7f/0x39506bfd0afc74b3 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.35@tcp remote: 0xa65ad49b6cb884b1 expref: 5 pid: 157308 timeout: 5145 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5045.601310] LustreError: lustre-MDT0000: A client on nid 192.168.204.35@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5045.623139] LustreError: 157296:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.35@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000003ca71d7f/0x39506bfd0afc74b3 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.35@tcp remote: 0xa65ad49b6cb884b1 expref: 6 pid: 157308 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5051.360630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5051.370911] LustreError: Skipped 1 previous similar message [ 5057.385819] Lustre: server umount lustre-OST0000 complete [ 5067.573163] Lustre: server umount lustre-MDT0000 complete [ 5071.665690] LustreError: 157292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225080 with bad export cookie 4129919592853958648 [ 5071.678281] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5071.685706] LustreError: 157292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5078.237954] Lustre: server umount lustre-MDT0001 complete [ 5089.927655] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5093.549989] Key type lgssc unregistered [ 5093.889771] LNet: 163416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5093.907803] LNetError: 163416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5093.927689] LNet: Removed LNI 192.168.204.135@tcp [ 5094.984505] Key type .llcrypt unregistered [ 5094.987083] Key type ._llcrypt unregistered [ 5124.269989] Key type ._llcrypt registered [ 5124.276803] Key type .llcrypt registered [ 5124.391407] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5125.775384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5125.822525] alg: No test for adler32 (adler32-zlib) [ 5126.925417] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 5127.236903] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5128.940225] Key type lgssc registered [ 5130.490808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5137.110357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5137.121202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.537499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5143.751077] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5149.154085] LustreError: 164844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.175134] LustreError: 164844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5154.281564] LustreError: 164845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.308385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.638431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.904300] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5170.416235] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5177.007272] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5185.656198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5186.044789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.532631] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5202.394575] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5209.654969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5217.877515] Lustre: Failing over lustre-MDT0000 [ 5218.183985] Lustre: server umount lustre-MDT0000 complete [ 5218.248755] LustreError: 166864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.814815] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.835751] Lustre: Skipped 1 previous similar message [ 5224.937355] LustreError: 164844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.952107] LustreError: 164844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5230.052870] LustreError: 166864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.069637] LustreError: 166864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5230.711317] LustreError: 164824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.35@tcp arrived at 1782225239 with bad export cookie 6068125683601277528 [ 5235.169967] Lustre: 164157:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782225228/real 1782225228] req@00000000a09dfe57 x1868798504237312/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782225244 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5235.207480] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5238.468541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.298759] LustreError: 164840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.326547] LustreError: 164840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5245.643542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.699717] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5250.333312] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5251.050371] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5251.056917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5252.438056] Lustre: lustre-MDT0000: Denying connection for new client 5290ecb2-dc42-443f-b606-6320c4c5e60b (at 192.168.204.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5261.283421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.284622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5261.292485] Lustre: Skipped 2 previous similar messages [ 5261.298722] Lustre: Skipped 1 previous similar message [ 5266.403981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.422666] Lustre: Skipped 1 previous similar message [ 5266.910083] Lustre: server umount lustre-OST0000 complete [ 5271.004421] LustreError: 168599:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5271.014441] Lustre: 167949:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5271.028322] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5271.052826] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5271.064610] LustreError: 167949:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a036741a x1868798504256256/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5271.084460] LustreError: 167949:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5271.095968] LustreError: 167949:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5271.108201] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5271.493459] Lustre: server umount lustre-MDT0000 complete [ 5271.531600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.552887] LustreError: 166864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.577910] LustreError: 166864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5275.509219] LustreError: 166688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225284 with bad export cookie 6068125683601277850 [ 5275.513523] LustreError: 166688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5275.515154] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.517363] LustreError: 164153:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5277.850608] Lustre: server umount lustre-MDT0001 complete [ 5290.034445] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5292.257632] Key type lgssc unregistered [ 5292.549693] LNet: 169351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5292.569303] LNetError: 169351:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5292.591072] LNet: Removed LNI 192.168.204.135@tcp [ 5293.328824] Key type .llcrypt unregistered [ 5293.331118] Key type ._llcrypt unregistered [ 5317.831781] Key type ._llcrypt registered [ 5317.835440] Key type .llcrypt registered [ 5317.935163] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5318.898276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5318.947703] alg: No test for adler32 (adler32-zlib) [ 5320.057956] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 5320.314751] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5321.967204] Key type lgssc registered [ 5323.072581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5331.798311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5331.814869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.248038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.894301] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5343.733246] LustreError: 170763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.760087] LustreError: 170763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5346.545510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.215298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5352.747944] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5362.980245] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5369.305865] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5376.507926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5376.756745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.818701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5384.193773] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5391.777204] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5399.520186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.530751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.541315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5402.592760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5402.598542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5402.607500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.436547] Lustre: server umount lustre-OST0000 complete [ 5412.834126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.835793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.854637] Lustre: Skipped 1 previous similar message [ 5412.872228] Lustre: Skipped 2 previous similar messages [ 5415.798214] Lustre: server umount lustre-MDT0000 complete [ 5417.956374] LustreError: 170759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.979821] LustreError: 170759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5419.777822] LustreError: 171499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225428 with bad export cookie 7206331756452679 [ 5419.781513] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5419.793876] LustreError: 171499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5420.173982] Lustre: server umount lustre-MDT0001 complete [ 5431.467290] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5434.278942] Key type lgssc unregistered [ 5434.545218] LNet: 174296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5434.564780] LNetError: 174296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5434.589749] LNet: Removed LNI 192.168.204.135@tcp [ 5435.361631] Key type .llcrypt unregistered [ 5435.364516] Key type ._llcrypt unregistered [ 5469.491201] Key type ._llcrypt registered [ 5469.494506] Key type .llcrypt registered [ 5469.596370] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5470.691546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5470.787985] alg: No test for adler32 (adler32-zlib) [ 5471.858428] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 5472.131915] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5473.823246] Key type lgssc registered [ 5474.918740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5482.872660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5482.887056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.487583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.602897] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5494.755942] LustreError: 175999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5494.773985] LustreError: 175999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5498.093963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5498.417398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5502.866676] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5511.119413] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5518.065533] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5524.560239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5524.830600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.986179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5531.794509] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5540.311686] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5558.561792] Lustre: server umount lustre-OST0000 complete [ 5563.872670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.886646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5563.896039] Lustre: Skipped 1 previous similar message [ 5563.907481] Lustre: Skipped 1 previous similar message [ 5568.996820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.008620] Lustre: Skipped 1 previous similar message [ 5573.238178] Lustre: server umount lustre-MDT0000 complete [ 5574.118811] LustreError: 176727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.138848] LustreError: 176727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5575.135639] Lustre: 175309:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782225567/real 1782225567] req@00000000c24613ef x1868798864946560/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782225583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5575.138050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.174240] Lustre: 175309:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5575.194676] LustreError: 175999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.214146] LustreError: 175981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225586 with bad export cookie 11342237360973457782 [ 5577.224107] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5577.633793] Lustre: server umount lustre-MDT0001 complete [ 5589.698254] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5592.470815] Key type lgssc unregistered [ 5592.903502] LNet: 179557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5592.910263] LNetError: 179557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5592.925807] LNet: Removed LNI 192.168.204.135@tcp [ 5594.055927] Key type .llcrypt unregistered [ 5594.058265] Key type ._llcrypt unregistered [ 5616.877363] Key type ._llcrypt registered [ 5616.880521] Key type .llcrypt registered [ 5616.983652] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5618.043492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5618.095113] alg: No test for adler32 (adler32-zlib) [ 5619.158972] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 5619.529636] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5621.239201] Key type lgssc registered [ 5622.232646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5633.122541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5633.147252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.531146] Lustre: *** cfs_fail_loc=135, val=0*** [ 5634.636870] LustreError: 181080:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5634.646837] LustreError: 181080:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -2 [ 5634.658192] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5634.658192] [ 5634.677087] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5634.700652] LustreError: 181041:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5634.719552] LustreError: 181041:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5634.737509] LustreError: 181041:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5634.747381] LustreError: 181041:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5634.852443] Lustre: server umount lustre-MDT0000 complete [ 5634.857755] LustreError: 181041:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5641.447974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.861461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.008400] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5651.427968] LustreError: 181451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.457226] LustreError: 181451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5656.548495] LustreError: 181452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.868608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.284271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5662.307554] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5669.917594] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5675.153280] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5678.054535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.057100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5678.068135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5678.079152] Lustre: Skipped 1 previous similar message [ 5682.211052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.219993] Lustre: Skipped 1 previous similar message [ 5687.270748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5687.288160] Lustre: Skipped 1 previous similar message [ 5691.359643] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5691.638668] Lustre: server umount lustre-MDT0000 complete [ 5692.386694] LustreError: 181452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.445944] LustreError: 181452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5695.830067] LustreError: 183021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225704 with bad export cookie 15101281527228366809 [ 5695.844304] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.845992] LustreError: 183021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5696.445845] Lustre: server umount lustre-MDT0001 complete [ 5710.626728] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5714.380826] Key type lgssc unregistered [ 5714.807722] LNet: 183783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5714.822651] LNetError: 183783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5714.883175] LNet: Removed LNI 192.168.204.135@tcp [ 5716.234776] Key type .llcrypt unregistered [ 5716.237648] Key type ._llcrypt unregistered [ 5741.135976] Key type ._llcrypt registered [ 5741.138415] Key type .llcrypt registered [ 5741.266188] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5741.644650] Key type .llcrypt unregistered [ 5741.650666] Key type ._llcrypt unregistered [ 5759.352966] Key type ._llcrypt registered [ 5759.359214] Key type .llcrypt registered [ 5759.483822] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5760.969090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5761.011533] alg: No test for adler32 (adler32-zlib) [ 5762.135209] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 5762.346541] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5764.031212] Key type lgssc registered [ 5765.565489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5774.646910] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5774.676743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5790.239708] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5795.359520] LustreError: 185919:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 5822.047970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5828.381460] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5835.703685] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5842.874051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.587160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.695284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5848.291739] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5849.081431] LustreError: 185944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.107784] LustreError: 185944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5854.181248] LustreError: 185945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.244571] LustreError: 185944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.268452] LustreError: 185944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5857.278895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.707701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5861.558124] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5868.664259] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5874.571828] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5877.842043] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5893.603589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5893.603963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.619231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5893.636285] Lustre: Skipped 1 previous similar message [ 5893.649071] Lustre: Skipped 1 previous similar message [ 5896.575784] Lustre: server umount lustre-OST0000 complete [ 5903.840499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.842910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.860886] Lustre: Skipped 1 previous similar message [ 5903.877555] Lustre: Skipped 1 previous similar message [ 5906.581944] Lustre: server umount lustre-MDT0000 complete [ 5908.962754] LustreError: 186961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.985732] LustreError: 186961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5910.640080] LustreError: 185925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225919 with bad export cookie 2843531202425131387 [ 5910.643752] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.652389] LustreError: 185925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5911.034835] Lustre: server umount lustre-MDT0001 complete [ 5923.178334] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5926.045670] Key type lgssc unregistered [ 5926.362121] LNet: 189933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5926.380967] LNetError: 189933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5927.464717] LNet: Removed LNI 192.168.204.135@tcp [ 5928.408060] Key type .llcrypt unregistered [ 5928.412360] Key type ._llcrypt unregistered [ 5951.689930] Key type ._llcrypt registered [ 5951.694802] Key type .llcrypt registered [ 5951.791423] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5952.891819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5952.919862] alg: No test for adler32 (adler32-zlib) [ 5954.003451] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 5954.221588] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5955.967165] Key type lgssc registered [ 5957.043617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.654886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5964.670217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.114934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.328802] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5975.524534] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.553972] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5979.649033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.072535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.224465] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5990.855729] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5996.487261] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6003.359160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6003.654287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.359552] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6010.879239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6018.222131] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6022.891504] Lustre: Failing over lustre-MDT0000 [ 6023.141107] Lustre: server umount lustre-MDT0000 complete [ 6025.162245] LustreError: 191338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.210481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6026.214628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.233036] LustreError: 191340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.252420] Lustre: Skipped 2 previous similar messages [ 6026.288449] LustreError: 191340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6030.243577] LustreError: 191338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.360979] LustreError: 191338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.393767] LustreError: 191338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6042.289499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6042.439317] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.611095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.666236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6045.605892] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6047.324301] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6047.740693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6047.824751] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6047.897593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6053.834213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6061.388468] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6073.313661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.342127] Lustre: Skipped 1 previous similar message [ 6073.349978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.434188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.445079] Lustre: Skipped 2 previous similar messages [ 6079.282854] Lustre: server umount lustre-OST0000 complete [ 6083.554440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.555411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.561391] Lustre: Skipped 1 previous similar message [ 6088.680551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6088.684631] Lustre: Skipped 2 previous similar messages [ 6089.533825] Lustre: server umount lustre-MDT0000 complete [ 6093.471991] LustreError: 194925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226102 with bad export cookie 920800291117949553 [ 6093.480635] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6093.482880] LustreError: 194925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6093.844269] Lustre: server umount lustre-MDT0001 complete [ 6106.730996] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6109.968415] Key type lgssc unregistered [ 6110.317805] LNet: 196188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6110.328202] LNetError: 196188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6110.343905] LNet: Removed LNI 192.168.204.135@tcp [ 6111.241292] Key type .llcrypt unregistered [ 6111.245448] Key type ._llcrypt unregistered [ 6137.350086] Key type ._llcrypt registered [ 6137.353142] Key type .llcrypt registered [ 6137.453629] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6138.369976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6138.423973] alg: No test for adler32 (adler32-zlib) [ 6139.401354] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 6139.604526] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6141.248567] Key type lgssc registered [ 6142.253444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6149.760488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6149.783243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.255500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.283547] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6161.392443] LustreError: 197598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.417366] LustreError: 197598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6162.810157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.090894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6167.392127] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6174.692800] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6180.781506] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6187.307667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6187.732317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.292137] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6196.975732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6201.080605] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6204.742595] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6208.107896] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6223.558690] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6238.180250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.198862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6238.212914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6238.232788] Lustre: Skipped 1 previous similar message [ 6243.300475] LustreError: 199421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.344604] LustreError: 199421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6243.450377] Lustre: server umount lustre-OST0000 complete [ 6248.417818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.423915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6248.438506] Lustre: Skipped 2 previous similar messages [ 6248.453519] Lustre: Skipped 1 previous similar message [ 6253.550759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.560873] Lustre: Skipped 1 previous similar message [ 6258.656476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6258.668836] Lustre: Skipped 1 previous similar message [ 6262.239127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6262.478264] Lustre: server umount lustre-MDT0000 complete [ 6263.782360] LustreError: 198326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.823943] LustreError: 198326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6266.286371] LustreError: 198334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226275 with bad export cookie 12725219406035561798 [ 6266.289835] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.291807] LustreError: 198334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6266.605745] Lustre: server umount lustre-MDT0001 complete [ 6280.062616] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6283.301553] Key type lgssc unregistered [ 6283.620695] LNet: 201468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6283.639864] LNetError: 201468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6283.669199] LNet: Removed LNI 192.168.204.135@tcp [ 6284.540555] Key type .llcrypt unregistered [ 6284.555365] Key type ._llcrypt unregistered [ 6302.990583] Key type ._llcrypt registered [ 6302.992620] Key type .llcrypt registered [ 6303.082949] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6304.094587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6304.121205] alg: No test for adler32 (adler32-zlib) [ 6305.150890] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 6305.379798] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6307.096596] Key type lgssc registered [ 6308.408166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6316.654696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6316.675199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.158907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.939537] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6327.277070] LustreError: 202735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6327.305382] LustreError: 202735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6332.241234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6332.385922] LustreError: 202736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.702144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6337.190666] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6344.865891] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6352.059086] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6359.563072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.073645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.185374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6366.233463] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6374.587966] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6381.331338] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6395.125379] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6403.860718] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6411.502822] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6419.425193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.431123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.440978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.520314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6423.524423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.530696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.538649] Lustre: Skipped 1 previous similar message [ 6424.728973] Lustre: server umount lustre-OST0000 complete [ 6428.642910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6428.660125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6433.769826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6433.773477] Lustre: Skipped 2 previous similar messages [ 6434.392511] Lustre: server umount lustre-MDT0000 complete [ 6438.334535] LustreError: 202715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226447 with bad export cookie 4211934304881499136 [ 6438.341773] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6438.357082] LustreError: 202715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6438.752839] Lustre: server umount lustre-MDT0001 complete [ 6451.010128] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6453.896173] Key type lgssc unregistered [ 6454.148769] LNet: 207660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6454.159267] LNetError: 207660:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6454.178771] LNet: Removed LNI 192.168.204.135@tcp [ 6454.822376] Key type .llcrypt unregistered [ 6454.825750] Key type ._llcrypt unregistered [ 6480.527755] Key type ._llcrypt registered [ 6480.530187] Key type .llcrypt registered [ 6480.641428] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6481.605895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6481.745733] alg: No test for adler32 (adler32-zlib) [ 6482.894691] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 6483.191883] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6484.928329] Key type lgssc registered [ 6486.173146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6494.060651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6494.091969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.487189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.655069] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6505.957247] LustreError: 209072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.970548] LustreError: 209072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6509.359791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.904366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6515.041655] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6522.903980] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6529.098705] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6536.076074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6536.479400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.588709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6541.629276] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6549.923116] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6570.466574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.475649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6570.496568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6573.397223] Lustre: server umount lustre-OST0000 complete [ 6580.708087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.716673] Lustre: Skipped 1 previous similar message [ 6580.728596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.740124] Lustre: Skipped 2 previous similar messages [ 6583.126161] Lustre: server umount lustre-MDT0000 complete [ 6585.832912] LustreError: 210061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.851533] LustreError: 210061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6587.298686] LustreError: 209052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226596 with bad export cookie 2089193554573843669 [ 6587.300554] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6587.310711] LustreError: 209052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6587.771251] Lustre: server umount lustre-MDT0001 complete [ 6600.627456] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6603.017482] Key type lgssc unregistered [ 6603.296585] LNet: 212754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6603.310281] LNetError: 212754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6603.343854] LNet: Removed LNI 192.168.204.135@tcp [ 6604.306621] Key type .llcrypt unregistered [ 6604.310353] Key type ._llcrypt unregistered [ 6625.954692] Key type ._llcrypt registered [ 6625.962901] Key type .llcrypt registered [ 6626.214722] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6627.932732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6628.036889] alg: No test for adler32 (adler32-zlib) [ 6629.314596] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 6629.760411] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6631.471250] Key type lgssc registered [ 6633.232922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6641.482137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6641.519865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.867723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.820192] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6652.388841] LustreError: 214022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.421334] LustreError: 214022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6657.512645] LustreError: 214021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.740376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.192601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6662.872478] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6671.476370] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6678.056336] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6685.822086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6686.248375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.309587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6693.540121] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6702.175521] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6723.041296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6723.041979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.050694] LustreError: Skipped 1 previous similar message [ 6723.055805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6723.070713] Lustre: Skipped 1 previous similar message [ 6723.095994] Lustre: Skipped 1 previous similar message [ 6724.262301] Lustre: server umount lustre-OST0000 complete [ 6729.696687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6729.716863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.730263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.280434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.282149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.494915] Lustre: server umount lustre-MDT0000 complete [ 6738.325240] LustreError: 214002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226747 with bad export cookie 18328982808265253583 [ 6738.332836] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.341861] LustreError: 214002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6738.627519] Lustre: server umount lustre-MDT0001 complete [ 6750.037744] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6752.724740] Key type lgssc unregistered [ 6753.060535] LNet: 217706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6753.072206] LNetError: 217706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6754.090493] LNet: Removed LNI 192.168.204.135@tcp [ 6754.998933] Key type .llcrypt unregistered [ 6755.001156] Key type ._llcrypt unregistered [ 6774.834173] Key type ._llcrypt registered [ 6774.837526] Key type .llcrypt registered [ 6774.930246] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6775.875914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6776.028076] alg: No test for adler32 (adler32-zlib) [ 6777.045965] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 6777.437162] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6779.148262] Key type lgssc registered [ 6780.259937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6787.863344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6787.880650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.378930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6793.617199] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6799.842352] LustreError: 218974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6799.863258] LustreError: 218974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6802.963613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.401359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6808.472754] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6816.961218] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6825.230676] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6833.313464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.810774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.878784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6841.126626] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6849.550928] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6860.052820] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6860.130445] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6860.363630] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6867.488218] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6877.864904] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6879.981808] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6891.844824] Lustre: Permanently deactivating lustre-OST0001 [ 6891.860194] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6897.712413] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6905.020898] Lustre: Permanently reactivating lustre-OST0001 [ 6905.032612] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6905.038907] Lustre: Skipped 2 previous similar messages [ 6912.620152] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.637741] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6912.648674] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6912.664851] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6912.719555] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6916.618732] Lustre: server umount lustre-OST0001 complete [ 6917.607908] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.613973] LustreError: 220794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.619481] Lustre: Skipped 2 previous similar messages [ 6917.686542] LustreError: 220794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6922.210282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.218495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.243115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.726733] LustreError: 220797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.745101] LustreError: 220797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6926.738839] Lustre: server umount lustre-OST0000 complete [ 6932.961340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6932.963088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6932.973178] Lustre: Skipped 2 previous similar messages [ 6932.981537] Lustre: Skipped 2 previous similar messages [ 6938.080210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6938.088516] Lustre: Skipped 1 previous similar message [ 6942.181900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6942.190318] Lustre: Skipped 1 previous similar message [ 6946.271129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6946.494492] Lustre: server umount lustre-MDT0000 complete [ 6947.298745] LustreError: 218974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.315706] LustreError: 218974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6950.843642] LustreError: 218954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226959 with bad export cookie 15387188736372168017 [ 6950.853375] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6950.860604] LustreError: 218954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6951.254994] Lustre: server umount lustre-MDT0001 complete [ 6964.048310] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6967.242998] Key type lgssc unregistered [ 6967.623817] LNet: 224035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6967.631229] LNetError: 224035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6967.643709] LNet: Removed LNI 192.168.204.135@tcp [ 6968.512664] Key type .llcrypt unregistered [ 6968.521164] Key type ._llcrypt unregistered [ 7011.852216] Key type ._llcrypt registered [ 7011.853848] Key type .llcrypt registered [ 7011.988493] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7013.215539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7013.300894] alg: No test for adler32 (adler32-zlib) [ 7014.395895] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 7014.625609] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7016.281862] Key type lgssc registered [ 7017.345947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7024.704408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7024.719568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.087931] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7026.127732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7026.305433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.886516] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7040.691620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.863268] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7040.899715] Lustre: 226242:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7041.175062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.948114] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7053.891740] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7060.112301] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7067.431257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7067.662962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7067.896711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.092393] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7076.552081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7083.758350] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7087.666964] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7093.278397] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7103.597590] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7114.007112] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7128.765314] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7128.770460] Lustre: Skipped 1 previous similar message [ 7146.103716] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7146.108476] Lustre: Skipped 1 previous similar message [ 7185.417882] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7185.428848] Lustre: Skipped 4 previous similar messages [ 7252.443815] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7252.449667] Lustre: Skipped 8 previous similar messages [ 7261.169782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7261.177881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.198675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7261.220616] Lustre: Skipped 1 previous similar message [ 7261.241438] Lustre: Skipped 1 previous similar message [ 7266.280420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7266.298118] Lustre: Skipped 1 previous similar message [ 7266.470921] Lustre: server umount lustre-OST0000 complete [ 7271.413250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.422340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.440181] Lustre: Skipped 1 previous similar message [ 7276.512524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7276.519470] Lustre: Skipped 2 previous similar messages [ 7276.733074] Lustre: server umount lustre-MDT0000 complete [ 7281.213581] LustreError: 226982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227290 with bad export cookie 5419106320870202531 [ 7281.216834] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7281.223097] LustreError: 226982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7281.693306] Lustre: server umount lustre-MDT0001 complete [ 7293.733478] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7296.789277] Key type lgssc unregistered [ 7297.137591] LNet: 231043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7297.149232] LNetError: 231043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7297.173179] LNet: Removed LNI 192.168.204.135@tcp [ 7298.058751] Key type .llcrypt unregistered [ 7298.062759] Key type ._llcrypt unregistered [ 7324.390538] Key type ._llcrypt registered [ 7324.398145] Key type .llcrypt registered [ 7324.510325] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7325.672950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7325.717045] alg: No test for adler32 (adler32-zlib) [ 7326.800881] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 7326.993609] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7328.664090] Key type lgssc registered [ 7329.682682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7337.192333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7337.210340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7338.675963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.801542] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7348.195344] LustreError: 232459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.238032] LustreError: 232459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7353.323538] LustreError: 232458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7353.478236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.941082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7358.646176] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7366.394322] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7371.986464] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7378.894694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7379.329717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7383.427392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7386.016570] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7393.288338] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7409.134781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.162055] Lustre: Skipped 1 previous similar message [ 7409.167137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7409.174821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7413.470566] Lustre: server umount lustre-OST0000 complete [ 7419.362562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.368902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.391418] Lustre: Skipped 1 previous similar message [ 7419.407334] Lustre: Skipped 2 previous similar messages [ 7424.482996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7424.490683] Lustre: Skipped 1 previous similar message [ 7429.623129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7429.629934] Lustre: Skipped 1 previous similar message [ 7432.159145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7432.627897] Lustre: server umount lustre-MDT0000 complete [ 7434.721127] LustreError: 232459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.750438] LustreError: 232459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7437.371141] LustreError: 232440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227446 with bad export cookie 4143850548189651181 [ 7437.373288] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7437.390178] LustreError: 232440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7437.965596] Lustre: server umount lustre-MDT0001 complete [ 7451.151433] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7454.063029] Key type lgssc unregistered [ 7454.349325] LNet: 236148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7454.357207] LNetError: 236148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7454.380292] LNet: Removed LNI 192.168.204.135@tcp [ 7455.346113] Key type .llcrypt unregistered [ 7455.349210] Key type ._llcrypt unregistered [ 7481.380892] Key type ._llcrypt registered [ 7481.383064] Key type .llcrypt registered [ 7481.549129] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7482.750633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7482.822116] alg: No test for adler32 (adler32-zlib) [ 7483.846559] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 7484.072698] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7485.762283] Key type lgssc registered [ 7486.876694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7760.896484] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7763.474397] Key type lgssc unregistered [ 7763.800505] LNet: 238172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7763.805676] LNetError: 238172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7763.821770] LNet: Removed LNI 192.168.204.135@tcp [ 7764.598684] Key type .llcrypt unregistered [ 7764.603782] Key type ._llcrypt unregistered [ 7791.323409] Key type ._llcrypt registered [ 7791.330251] Key type .llcrypt registered [ 7791.511890] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 7809.692619] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7810.785981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7810.838640] alg: No test for adler32 (adler32-zlib) [ 7811.902840] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 7812.161586] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7813.863173] Key type lgssc registered [ 7815.068045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7859.175788] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7597 sec ========== 11:17:46 (1782227866) [ 7861.246932] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:17:48 (1782227868) === [ 7865.003249] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:17:52 (1782227872) === [ 7909.074636] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7911.831192] Key type lgssc unregistered [ 7912.154682] LNet: 243413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7912.166692] LNetError: 243413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7912.208191] LNet: Removed LNI 192.168.204.135@tcp [ 7913.087857] Key type .llcrypt unregistered [ 7913.089915] Key type ._llcrypt unregistered