[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488071715 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007416] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011107] LSM: Security Framework initializing [ 0.012045] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.022487] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026440] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029097] *** VALIDATE tmpfs *** [ 0.030423] *** VALIDATE proc *** [ 0.032168] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.041494] debug: unmapping init [mem 0xffffffffbba59000-0xffffffffbba60fff] [ 0.043837] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044654] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045023] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052250] rcu: Hierarchical SRCU implementation. [ 0.054385] smp: Bringing up secondary CPUs ... [ 0.055517] x86: Booting SMP configuration: [ 0.056022] .... node #0, CPUs: #1 #2 #3 [ 0.064376] smp: Brought up 1 node, 4 CPUs [ 0.066013] smpboot: Max logical packages: 1 [ 0.067012] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.129102] node 0 deferred pages initialised in 60ms [ 0.133135] devtmpfs: initialized [ 0.134257] x86/mm: Memory block size: 128MB [ 0.137099] gcov: version magic: 0x41383552 [ 0.139308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141229] pinctrl core: initialized pinctrl subsystem [ 0.142294] [ 0.142951] ************************************************************* [ 0.143015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144010] ** ** [ 0.145013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146015] ** ** [ 0.147015] ** This means that this kernel is built to expose internal ** [ 0.148012] ** IOMMU data structures, which may compromise security on ** [ 0.149014] ** your system. ** [ 0.150013] ** ** [ 0.151014] ** If you see this message and you are not debugging the ** [ 0.152011] ** kernel, report this immediately to your vendor! ** [ 0.153011] ** ** [ 0.154012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155011] ************************************************************* [ 0.156770] NET: Registered protocol family 16 [ 0.157421] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.162049] cpuidle: using governor menu [ 0.163801] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.167615] PCI: Using configuration type 1 for base access [ 0.170127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181040] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184033] cryptd: max_cpu_qlen set to 1000 [ 0.187226] ACPI: Added _OSI(Module Device) [ 0.189018] ACPI: Added _OSI(Processor Device) [ 0.191012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199923] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207654] ACPI: Interpreter enabled [ 0.210116] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212016] ACPI: Using IOAPIC for interrupt routing [ 0.214156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217506] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228741] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241232] acpiphp: Slot [2] registered [ 0.242095] acpiphp: Slot [5] registered [ 0.244115] acpiphp: Slot [6] registered [ 0.245187] acpiphp: Slot [7] registered [ 0.247090] acpiphp: Slot [8] registered [ 0.248102] acpiphp: Slot [9] registered [ 0.249085] acpiphp: Slot [10] registered [ 0.250096] acpiphp: Slot [3] registered [ 0.252076] acpiphp: Slot [4] registered [ 0.253068] acpiphp: Slot [11] registered [ 0.254080] acpiphp: Slot [12] registered [ 0.255089] acpiphp: Slot [13] registered [ 0.256085] acpiphp: Slot [14] registered [ 0.257082] acpiphp: Slot [15] registered [ 0.258074] acpiphp: Slot [16] registered [ 0.260073] acpiphp: Slot [17] registered [ 0.261070] acpiphp: Slot [18] registered [ 0.262156] acpiphp: Slot [19] registered [ 0.263069] acpiphp: Slot [20] registered [ 0.264071] acpiphp: Slot [21] registered [ 0.265073] acpiphp: Slot [22] registered [ 0.266068] acpiphp: Slot [23] registered [ 0.267070] acpiphp: Slot [24] registered [ 0.269069] acpiphp: Slot [25] registered [ 0.270152] acpiphp: Slot [26] registered [ 0.271071] acpiphp: Slot [27] registered [ 0.272073] acpiphp: Slot [28] registered [ 0.273067] acpiphp: Slot [29] registered [ 0.274072] acpiphp: Slot [30] registered [ 0.275083] acpiphp: Slot [31] registered [ 0.276052] PCI host bridge to bus 0000:00 [ 0.277015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.279016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.281016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288130] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290832] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.293037] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.302706] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.309059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.312019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321494] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323644] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327696] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.347017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.352014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.358078] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.366014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.375013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.394024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.404838] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.412015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.483224] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.491018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.498016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.534697] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.542014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.577015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590625] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.599015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.607016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645211] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.676020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.696022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.707602] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710467] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.713608] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.716442] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.720272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.725134] iommu: Default domain type: Passthrough [ 0.726376] SCSI subsystem initialized [ 0.727109] ACPI: bus type USB registered [ 0.728110] usbcore: registered new interface driver usbfs [ 0.730074] usbcore: registered new interface driver hub [ 0.732079] usbcore: registered new device driver usb [ 0.733147] pps_core: LinuxPPS API ver. 1 registered [ 0.735011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.738079] PTP clock support registered [ 0.740084] EDAC MC: Ver: 3.0.0 [ 0.742174] PCI: Using ACPI for IRQ routing [ 0.743780] NetLabel: Initializing [ 0.744012] NetLabel: domain hash size = 128 [ 0.745010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.746129] NetLabel: unlabeled traffic allowed by default [ 0.748099] vgaarb: loaded [ 0.749207] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.752015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.759796] clocksource: Switched to clocksource kvm-clock [ 0.881941] VFS: Disk quotas dquot_6.6.0 [ 0.884189] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886705] *** VALIDATE ramfs *** [ 0.887964] *** VALIDATE hugetlbfs *** [ 0.889406] pnp: PnP ACPI init [ 0.891393] pnp: PnP ACPI: found 6 devices [ 0.909915] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.913858] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.916633] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.919388] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.922137] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.924890] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.928364] NET: Registered protocol family 2 [ 0.930990] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.937075] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.941192] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947451] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.951475] TCP: Hash tables configured (established 65536 bind 65536) [ 0.955055] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.958672] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.962465] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967273] NET: Registered protocol family 1 [ 0.971311] RPC: Registered named UNIX socket transport module. [ 0.973073] RPC: Registered udp transport module. [ 0.975044] RPC: Registered tcp transport module. [ 0.977307] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.980339] NET: Registered protocol family 44 [ 0.982238] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.984736] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.987313] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.989762] PCI: CLS 0 bytes, default 64 [ 0.991695] Unpacking initramfs... [ 2.440786] debug: unmapping init [mem 0xffff9db43cc54000-0xffff9db43ffbffff] [ 2.445129] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.449759] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.452473] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.956569] Initialise system trusted keyrings [ 2.958181] Key type blacklist registered [ 2.960640] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.971258] zbud: loaded [ 2.974783] *** VALIDATE nfs *** [ 2.976320] *** VALIDATE nfs4 *** [ 2.977894] pstore: using deflate compression [ 2.980960] Platform Keyring initialized [ 3.107925] NET: Registered protocol family 38 [ 3.110016] Key type asymmetric registered [ 3.111628] Asymmetric key parser 'x509' registered [ 3.113642] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.117173] io scheduler mq-deadline registered [ 3.118794] io scheduler kyber registered [ 3.120623] io scheduler bfq registered [ 3.122611] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.125775] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.129441] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.132988] ACPI: Power Button [PWRF] [ 3.139520] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.147413] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.166828] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.178739] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.198737] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.228921] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.259334] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.265353] Non-volatile memory driver v1.3 [ 3.267647] Linux agpgart interface v0.103 [ 3.299500] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.302655] vda: detected capacity change from 0 to 77115392 [ 3.323341] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.326077] vdb: detected capacity change from 0 to 1073741824 [ 3.342700] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.345797] vdc: detected capacity change from 0 to 2621440000 [ 3.360926] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.364191] vdd: detected capacity change from 0 to 2621440000 [ 3.385946] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.389515] vde: detected capacity change from 0 to 4294967296 [ 3.405370] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408441] vdf: detected capacity change from 0 to 4294967296 [ 3.414788] libphy: Fixed MDIO Bus: probed [ 3.425675] usbcore: registered new interface driver usbserial_generic [ 3.428255] usbserial: USB Serial support registered for generic [ 3.430584] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.435590] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.437660] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.440252] mousedev: PS/2 mouse device common for all mice [ 3.445434] rtc_cmos 00:05: RTC can wake from S4 [ 3.448851] rtc_cmos 00:05: registered as rtc0 [ 3.451269] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.454629] intel_pstate: CPU model not supported [ 3.456947] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.458323] hid: raw HID events driver (C) Jiri Kosina [ 3.468670] usbcore: registered new interface driver usbhid [ 3.471565] usbhid: USB HID core driver [ 3.473395] drop_monitor: Initializing network drop monitor service [ 3.474572] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.480265] Initializing XFRM netlink socket [ 3.480663] NET: Registered protocol family 10 [ 3.483200] Segment Routing with IPv6 [ 3.487142] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.488091] NET: Registered protocol family 17 [ 3.496928] mpls_gso: MPLS GSO support [ 3.503347] RAS: Correctable Errors collector initialized. [ 3.505751] AVX version of gcm_enc/dec engaged. [ 3.507699] AES CTR mode by8 optimization enabled [ 3.594602] sched_clock: Marking stable (3594570750, 0)->(4529578958, -935008208) [ 3.598418] registered taskstats version 1 [ 3.601286] Loading compiled-in X.509 certificates [ 3.603260] zswap: loaded using pool lzo/zbud [ 3.628267] Key type big_key registered [ 3.640071] Key type encrypted registered [ 3.641884] ima: No TPM chip found, activating TPM-bypass! [ 3.644287] ima: Allocated hash algorithm: sha1 [ 3.646118] ima: No architecture policies found [ 3.647814] evm: Initialising EVM extended attributes: [ 3.649959] evm: security.selinux [ 3.651623] evm: security.ima [ 3.652788] evm: security.capability [ 3.654299] evm: HMAC attrs: 0x1 [ 3.656703] rtc_cmos 00:05: setting system clock to 2026-10-02 12:34:12 UTC (1790944452) [ 3.663502] debug: unmapping init [mem 0xffffffffbca03000-0xffffffffbcbfffff] [ 3.666895] debug: unmapping init [mem 0xffffffffbb782000-0xffffffffbba58fff] [ 3.679272] Write protecting the kernel read-only data: 28672k [ 3.683111] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 3.686143] debug: unmapping init [mem 0xffffffffba714000-0xffffffffba7fffff] [ 3.726693] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.736408] systemd[1]: Detected virtualization kvm. [ 3.738259] systemd[1]: Detected architecture x86-64. [ 3.740428] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.768612] systemd[1]: No hostname configured. [ 3.770340] systemd[1]: Set hostname to . [ 3.772707] random: systemd: uninitialized urandom read (16 bytes read) [ 3.775320] systemd[1]: Initializing machine ID from random generator. [ 3.932591] random: systemd: uninitialized urandom read (16 bytes read) [ 3.938652] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.947569] random: systemd: uninitialized urandom read (16 bytes read) [ 3.950419] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.955267] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.667758] device-mapper: uevent: version 1.0.3 [ 4.670674] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.568768] virtio_net virtio0 ens2: renamed from eth0 [ 5.575867] random: fast init done Starting dracut initqueue hook... [ 5.748327] scsi host0: ata_piix [ 5.759691] scsi host1: ata_piix [ 5.761552] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.763566] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.410512] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.163619] random: crng init done [ 10.165372] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.778210] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.084270] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.410167] SELinux: Disabled at runtime. [ 12.474985] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.484191] systemd[1]: Detected virtualization kvm. [ 12.485572] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.095498] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.101188] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.105972] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.111343] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.115400] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.122448] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.131585] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 13.275233] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.645462] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.985611] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.017598] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.148410] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.169275] EDAC sbridge: Ver: 1.1.2 [ 16.150075] Key type dns_resolver registered [ 16.470502] NFS: Registering the id_resolver key type [ 16.473637] Key type id_resolver registered [ 16.475342] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-server login: [ 61.237888] libcfs: loading out-of-tree module taints kernel. [ 61.345895] Key type ._llcrypt registered [ 61.351187] Key type .llcrypt registered [ 61.569873] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 86.067705] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 87.067905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.168419] alg: No test for adler32 (adler32-zlib) [ 88.678157] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 89.745920] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 91.535222] Key type lgssc registered [ 93.375939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.604135] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.846374] hrtimer: interrupt took 4355466 ns [ 157.908161] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 169.978655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 170.023594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.345541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.387554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.461981] Lustre: lustre-MDT0000: new disk, initializing [ 171.622268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.654053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.050424] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.016397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.162411] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 197.238893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 197.246954] Lustre: Skipped 1 previous similar message [ 197.366987] Lustre: lustre-MDT0001: new disk, initializing [ 197.474090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.529720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 197.551048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.411855] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.171867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.806876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.181874] Lustre: lustre-OST0000: new disk, initializing [ 220.186239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.191464] Lustre: 8447:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.267609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.380939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.391754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.450215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 225.269750] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.029995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.344475] Lustre: lustre-OST0001: new disk, initializing [ 241.348589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 241.362364] Lustre: 9518:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 241.502165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.483288] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.064937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.096467] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.176503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 264.401411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.789165] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.219832] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing check_logdir /tmp/testlogs/ [ 282.174363] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing yml_node [ 286.547717] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 289.500963] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 291.921630] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 293.543786] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 08:38:59 EDT 2026 [ 310.295302] Lustre: DEBUG MARKER: only running tests: 0-31 [ 311.993650] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 313.556694] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.721977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 330.732366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.759819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.284393] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.285432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.303129] Lustre: Skipped 2 previous similar messages [ 333.309086] Lustre: Skipped 2 previous similar messages [ 335.371521] Lustre: server umount lustre-MDT0000 complete [ 339.114402] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790944787 with bad export cookie 12970393954914586511 [ 339.120125] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.126601] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.519793] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.520061] LustreError: 8065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.522849] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.522859] Lustre: Skipped 1 previous similar message [ 343.537539] Lustre: Skipped 1 previous similar message [ 343.551565] LustreError: 8065:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 344.544189] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.599495] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 345.555858] Lustre: server umount lustre-MDT0001 complete [ 349.627967] Lustre: server umount lustre-OST0000 complete [ 353.239313] Lustre: server umount lustre-OST0001 complete [ 359.996689] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 368.318402] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 401.582788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.846107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.891555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.942223] Lustre: lustre-MDT0000: new disk, initializing [ 402.086425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.122692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.795081] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 417.645632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.879348] Lustre: 15118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.988705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.995762] Lustre: Skipped 1 previous similar message [ 418.106529] Lustre: lustre-MDT0001: new disk, initializing [ 418.279598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.380588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.404730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 423.814384] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 434.015616] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.796515] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 446.387252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 446.778728] Lustre: lustre-OST0000: new disk, initializing [ 446.783628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.792881] Lustre: 17088:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 446.907403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.558684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.571512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.759154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.104601] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 461.777805] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 465.705616] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.898486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.018561] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 470.210418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.599455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.605526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.614890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.617111] Lustre: Skipped 1 previous similar message [ 474.633045] Lustre: Skipped 4 previous similar messages [ 479.734453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.749791] Lustre: Skipped 1 previous similar message [ 486.369177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 486.528765] Lustre: server umount lustre-OST0000 complete [ 489.953221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.959577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.960891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.960897] Lustre: Skipped 2 previous similar messages [ 489.970587] Lustre: Skipped 1 previous similar message [ 489.978078] LustreError: Skipped 1 previous similar message [ 496.117415] Lustre: server umount lustre-MDT0000 complete [ 499.653637] LustreError: 15109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790944948 with bad export cookie 12970393954914588044 [ 499.658061] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.672421] LustreError: 15109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 500.028504] Lustre: server umount lustre-MDT0001 complete [ 503.353643] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 08:42:29 (1790944949) [ 509.186864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.665732] LustreError: 19352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.763786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.298537] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 515.040952] LustreError: 19353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.161953] LustreError: 19352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.679838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.008809] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 532.715496] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.844279] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 543.795458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 549.990764] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 556.998925] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 565.736688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.737143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.749425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.764349] Lustre: Skipped 1 previous similar message [ 565.783786] Lustre: Skipped 4 previous similar messages [ 570.092498] Lustre: server umount lustre-OST0000 complete [ 573.408343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.534718] Lustre: server umount lustre-MDT0000 complete [ 581.092289] LustreError: 19353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.118642] LustreError: 19353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 582.435961] LustreError: 21194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945031 with bad export cookie 12970393954914588926 [ 582.438236] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.457100] LustreError: 21194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 582.695148] Lustre: server umount lustre-MDT0001 complete [ 591.779709] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 593.681236] Key type lgssc unregistered [ 593.911913] LNet: 22882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.924549] LNetError: 22882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.943869] LNet: Removed LNI 192.168.203.149@tcp [ 594.743150] Key type .llcrypt unregistered [ 594.747950] Key type ._llcrypt unregistered [ 611.686828] Key type ._llcrypt registered [ 611.689058] Key type .llcrypt registered [ 611.770526] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 612.350750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.404596] alg: No test for adler32 (adler32-zlib) [ 613.347544] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 613.531772] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 615.175191] Key type lgssc registered [ 616.049894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 621.818993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 621.837867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.274950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.588554] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 632.479573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.752398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 636.063331] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 641.119928] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 645.015096] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 649.525449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 649.852849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.512685] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 660.104242] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 668.274556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 678.885957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.889361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 678.896167] Lustre: Skipped 1 previous similar message [ 678.896693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 678.908899] Lustre: Skipped 1 previous similar message [ 681.671404] Lustre: server umount lustre-OST0000 complete [ 689.122713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.123469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 689.127996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 689.131939] Lustre: Skipped 1 previous similar message [ 689.147435] LustreError: Skipped 1 previous similar message [ 690.724805] Lustre: server umount lustre-MDT0000 complete [ 693.448618] LustreError: 24270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945142 with bad export cookie 3025961546568833771 [ 693.452869] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.459986] LustreError: 24270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 693.679124] Lustre: server umount lustre-MDT0001 complete [ 701.714500] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 703.799905] Key type lgssc unregistered [ 704.020406] LNet: 28263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.028379] LNetError: 28263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.039760] LNet: Removed LNI 192.168.203.149@tcp [ 704.653720] Key type .llcrypt unregistered [ 704.655854] Key type ._llcrypt unregistered [ 723.497830] Key type ._llcrypt registered [ 723.501123] Key type .llcrypt registered [ 723.604818] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 724.416865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 724.465837] alg: No test for adler32 (adler32-zlib) [ 725.535683] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 725.737799] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 727.391209] Key type lgssc registered [ 728.362700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 734.807311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 734.839696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.256882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.894130] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 746.464264] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.482371] LustreError: 29670:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 747.821640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.172532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 752.706212] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 759.244258] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 764.094591] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 777.120332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.507367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.622176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 783.211342] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 789.895976] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 802.275960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.287767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.305459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 803.301728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 803.302716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.327178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 807.176845] Lustre: server umount lustre-OST0000 complete [ 813.542643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.543823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.555785] Lustre: Skipped 1 previous similar message [ 813.562655] Lustre: Skipped 1 previous similar message [ 817.516731] Lustre: server umount lustre-MDT0000 complete [ 818.656226] LustreError: 29671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.674723] LustreError: 29671:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 820.881151] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945269 with bad export cookie 2871100548619516540 [ 820.881845] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.891288] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 821.185938] Lustre: server umount lustre-MDT0001 complete [ 830.641727] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 833.500746] Key type lgssc unregistered [ 833.793775] LNet: 33547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.828857] LNetError: 33547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.855308] LNet: Removed LNI 192.168.203.149@tcp [ 834.574624] Key type .llcrypt unregistered [ 834.579221] Key type ._llcrypt unregistered [ 860.046206] Key type ._llcrypt registered [ 860.048032] Key type .llcrypt registered [ 860.115483] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 861.167448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.210153] alg: No test for adler32 (adler32-zlib) [ 862.473906] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 862.741523] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 864.431247] Key type lgssc registered [ 866.067301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 874.285683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 874.314931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.769154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.757494] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 885.218023] LustreError: 34978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.245504] LustreError: 34978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 888.785886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.091357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 893.214826] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 900.018375] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.693513] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 914.176234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 914.545950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.710880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 921.133820] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 927.987600] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 944.610085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 944.617650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.628476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.648389] Lustre: Skipped 1 previous similar message [ 948.003098] Lustre: server umount lustre-OST0000 complete [ 954.853565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 954.854802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 954.876333] Lustre: Skipped 1 previous similar message [ 954.894726] Lustre: Skipped 2 previous similar messages [ 957.633965] Lustre: server umount lustre-MDT0000 complete [ 959.986818] LustreError: 34978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.001858] LustreError: 34978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 961.340629] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945410 with bad export cookie 17556766009308490975 [ 961.353509] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.355546] LustreError: 34960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 961.746594] Lustre: server umount lustre-MDT0001 complete [ 974.128733] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 977.339602] Key type lgssc unregistered [ 977.724222] LNet: 38531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.740085] LNetError: 38531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.789466] LNet: Removed LNI 192.168.203.149@tcp [ 978.861830] Key type .llcrypt unregistered [ 978.863611] Key type ._llcrypt unregistered [ 1001.587437] Key type ._llcrypt registered [ 1001.591317] Key type .llcrypt registered [ 1001.769612] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1002.717303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1002.735669] alg: No test for adler32 (adler32-zlib) [ 1003.763450] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1003.989482] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1005.703856] Key type lgssc registered [ 1006.981738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.391640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1014.432731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.839334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.045338] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1026.017615] LustreError: 39960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.047829] LustreError: 39960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1028.030492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.344277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1032.545716] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1040.324426] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1047.157724] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1054.151131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1054.552923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.390305] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1060.725725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1067.246603] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1079.264359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.272740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.278400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.304102] Lustre: Skipped 1 previous similar message [ 1080.177110] Lustre: lustre-OST0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 1080.183678] Lustre: Skipped 1 previous similar message [ 1082.886834] Lustre: server umount lustre-OST0000 complete [ 1104.869370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.877049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.886455] Lustre: Skipped 1 previous similar message [ 1109.983894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.988018] Lustre: Skipped 1 previous similar message [ 1115.103145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1115.104469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.123491] Lustre: Skipped 2 previous similar messages [ 1115.267654] Lustre: server umount lustre-MDT0000 complete [ 1118.332758] LustreError: 39940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945567 with bad export cookie 16575357685067894698 [ 1118.345963] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.352164] LustreError: 39940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1124.687900] Lustre: server umount lustre-MDT0001 complete [ 1134.016441] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1136.342424] Key type lgssc unregistered [ 1136.590857] LNet: 43667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.596858] LNetError: 43667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.638396] LNet: Removed LNI 192.168.203.149@tcp [ 1138.305074] Key type .llcrypt unregistered [ 1138.310357] Key type ._llcrypt unregistered [ 1157.247438] Key type ._llcrypt registered [ 1157.251128] Key type .llcrypt registered [ 1157.329927] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1158.556178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1158.577960] alg: No test for adler32 (adler32-zlib) [ 1159.594079] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1159.821055] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1161.471261] Key type lgssc registered [ 1162.666259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1170.426449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1170.444584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.970168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.273507] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1182.181397] LustreError: 45074:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.188682] LustreError: 45074:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1184.116315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.446482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1188.662428] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1196.656562] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1202.485793] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1209.037634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.314805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.500451] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1217.540840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1223.183880] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1230.737676] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 1230.829634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.848177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.744308] Lustre: server umount lustre-MDT0000 complete [ 1232.864425] LustreError: 45074:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.882855] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.070560] LustreError: 45057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790945683 with bad export cookie 15278611218704092416 [ 1235.072042] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.085404] LustreError: 45057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1235.823603] LustreError: 45070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.854866] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 1235.862643] LustreError: 45070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1235.870645] Lustre: Skipped 1 previous similar message [ 1237.984235] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.986880] LustreError: 47198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.993647] Lustre: Skipped 1 previous similar message [ 1238.003521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1240.926546] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 1241.749633] Lustre: server umount lustre-MDT0001 complete [ 1263.052768] Lustre: server umount lustre-OST0000 complete [ 1277.962896] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1280.318414] Key type lgssc unregistered [ 1280.552314] LNet: 48932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.560041] LNetError: 48932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.585941] LNet: Removed LNI 192.168.203.149@tcp [ 1281.343121] Key type .llcrypt unregistered [ 1281.347793] Key type ._llcrypt unregistered [ 1301.201408] Key type ._llcrypt registered [ 1301.203126] Key type .llcrypt registered [ 1301.333471] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1302.112470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1302.143898] alg: No test for adler32 (adler32-zlib) [ 1303.190034] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1303.400064] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1305.031299] Key type lgssc registered [ 1306.023055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.437909] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1312.460178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.967960] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1333.088827] LustreError: 50329:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1359.711288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.024798] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1372.717695] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1538.086033] Lustre: server umount lustre-OST0000 complete [ 1551.953773] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1554.895805] Key type lgssc unregistered [ 1555.209954] LNet: 52003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.217812] LNetError: 52003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.238585] LNet: Removed LNI 192.168.203.149@tcp [ 1556.069226] Key type .llcrypt unregistered [ 1556.071209] Key type ._llcrypt unregistered [ 1575.274735] Key type ._llcrypt registered [ 1575.277940] Key type .llcrypt registered [ 1575.357979] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1576.083924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1576.103762] alg: No test for adler32 (adler32-zlib) [ 1577.117336] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1577.350595] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1579.007254] Key type lgssc registered [ 1580.120894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1587.994812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1588.022230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.630117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.287339] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1599.978542] LustreError: 53412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.007587] LustreError: 53412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1602.672020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.098503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.487937] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1613.877077] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1618.959822] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1624.822246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.175064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.621083] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1634.433377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1637.532723] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1639.907043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.910943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1639.926515] Lustre: Skipped 1 previous similar message [ 1641.500722] Lustre: server umount lustre-OST0000 complete [ 1645.024407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.036562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1645.045189] Lustre: Skipped 1 previous similar message [ 1645.055706] Lustre: Skipped 2 previous similar messages [ 1650.144790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.154326] Lustre: Skipped 1 previous similar message [ 1655.265551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.274575] Lustre: Skipped 1 previous similar message [ 1659.359131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1659.365775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.390306] Lustre: Skipped 1 previous similar message [ 1659.594240] Lustre: server umount lustre-MDT0000 complete [ 1662.781736] LustreError: 53392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946111 with bad export cookie 15580318926638474134 [ 1662.782231] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1662.787900] LustreError: 53392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1663.113500] Lustre: server umount lustre-MDT0001 complete [ 1672.227534] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1674.982119] Key type lgssc unregistered [ 1675.276940] LNet: 56956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.288471] LNetError: 56956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.314833] LNet: Removed LNI 192.168.203.149@tcp [ 1676.207053] Key type .llcrypt unregistered [ 1676.209875] Key type ._llcrypt unregistered [ 1696.550754] Key type ._llcrypt registered [ 1696.552554] Key type .llcrypt registered [ 1696.680762] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1697.471036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1697.545083] alg: No test for adler32 (adler32-zlib) [ 1698.544457] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1698.802374] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1700.455198] Key type lgssc registered [ 1701.518872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1710.084853] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1710.106893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.727635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1730.847576] LustreError: 58338:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1757.471153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.098837] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1768.279308] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1774.354574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.846603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.900448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1778.294225] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1780.197341] LustreError: 58363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.215955] LustreError: 58363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1785.313767] LustreError: 58364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.685606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.994203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1789.973323] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1796.075272] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1801.075766] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1805.793566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.805870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.822327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1806.304410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1806.314679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.329431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.844522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.164904] Lustre: server umount lustre-OST0000 complete [ 1831.904515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1831.912526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.926493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.931134] Lustre: Skipped 1 previous similar message [ 1837.461649] Lustre: server umount lustre-MDT0000 complete [ 1838.050630] LustreError: 59373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.071149] LustreError: 59373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1840.742453] LustreError: 61329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946289 with bad export cookie 6316057096871569452 [ 1840.752990] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.096876] Lustre: server umount lustre-MDT0001 complete [ 1855.814893] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1858.334547] Key type lgssc unregistered [ 1858.593722] LNet: 62080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.598865] LNetError: 62080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.628171] LNet: Removed LNI 192.168.203.149@tcp [ 1860.570270] Key type .llcrypt unregistered [ 1860.572391] Key type ._llcrypt unregistered [ 1881.895142] Key type ._llcrypt registered [ 1881.901720] Key type .llcrypt registered [ 1882.030438] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1883.037903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1883.131093] alg: No test for adler32 (adler32-zlib) [ 1884.292376] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 1884.475635] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1886.151196] Key type lgssc registered [ 1887.296225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.459464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1894.510713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.060367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.069261] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1905.656770] LustreError: 63515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.679475] LustreError: 63515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1909.322515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.672592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.351354] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1921.292525] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.184949] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1934.065645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1934.307304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.137136] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1940.480624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1948.018281] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1966.073539] Lustre: server umount lustre-OST0000 complete [ 1971.170790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.172762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1971.172917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1971.189954] Lustre: Skipped 1 previous similar message [ 1976.303546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.310633] Lustre: Skipped 1 previous similar message [ 1979.450128] Lustre: server umount lustre-MDT0000 complete [ 1981.412334] LustreError: 64775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.412640] Lustre: 62817:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790946414/real 1790946414] req@000000003c5e64b5 x1877943342101120/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790946430 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.434183] LustreError: 64775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1981.476023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.722349] LustreError: 63497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946431 with bad export cookie 2630638045049606999 [ 1982.738500] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.185478] Lustre: server umount lustre-MDT0001 complete [ 1994.291921] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1997.254039] Key type lgssc unregistered [ 1997.557769] LNet: 67067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.568097] LNetError: 67067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.582578] LNet: Removed LNI 192.168.203.149@tcp [ 1998.400202] Key type .llcrypt unregistered [ 1998.402536] Key type ._llcrypt unregistered [ 2025.971613] Key type ._llcrypt registered [ 2025.974188] Key type .llcrypt registered [ 2026.287201] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2027.986416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2028.162077] alg: No test for adler32 (adler32-zlib) [ 2029.275873] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2029.650761] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2031.312429] Key type lgssc registered [ 2032.851926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2041.932988] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:08:08 (1790946488) [ 2048.401866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2048.437391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.991814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.667199] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2059.241277] LustreError: 68788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.268759] LustreError: 68788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2064.165232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.354976] LustreError: 68789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.962937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.820137] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2078.207344] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.007979] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.415323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.795674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.907365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2095.649979] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2102.430700] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2112.068792] Lustre: Failing over lustre-MDT0000 [ 2112.419312] Lustre: server umount lustre-MDT0000 complete [ 2114.529068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.534275] LustreError: 68788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.536709] Lustre: Skipped 1 previous similar message [ 2114.562757] LustreError: 68788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2119.669753] LustreError: 68789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.690456] LustreError: 68789:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2120.984981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.075899] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.202254] Lustre: *** cfs_fail_loc=135, val=0*** [ 2121.288926] LustreError: 71877:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2121.299871] LustreError: 71877:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 2121.310084] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2121.310084] [ 2121.320394] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2121.333673] LustreError: 71865:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2121.341824] LustreError: 71865:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2121.349019] LustreError: 71865:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2121.354317] LustreError: 71865:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2121.381579] Lustre: server umount lustre-MDT0000 complete [ 2121.387843] LustreError: 71865:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2124.772118] LustreError: 69990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.798951] LustreError: 69990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2127.491517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.613667] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.915875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.991783] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2130.792261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2132.039565] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2132.981558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2133.010210] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2133.051249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2140.128171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2140.134057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.143392] Lustre: Skipped 1 previous similar message [ 2140.148084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.206506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.210579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.484803] Lustre: server umount lustre-OST0000 complete [ 2153.441831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2153.450192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.459864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.459872] Lustre: Skipped 1 previous similar message [ 2153.490647] Lustre: Skipped 1 previous similar message [ 2155.213597] Lustre: server umount lustre-MDT0000 complete [ 2158.565119] LustreError: 68789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.584612] LustreError: 68789:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2158.878713] LustreError: 68769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946607 with bad export cookie 17856212390846135933 [ 2158.884816] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2158.896881] LustreError: 68769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2159.336308] Lustre: server umount lustre-MDT0001 complete [ 2170.683894] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2174.342729] Key type lgssc unregistered [ 2174.798559] LNet: 73654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.810107] LNetError: 73654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.827166] LNet: Removed LNI 192.168.203.149@tcp [ 2175.780165] Key type .llcrypt unregistered [ 2175.781987] Key type ._llcrypt unregistered [ 2200.745648] Key type ._llcrypt registered [ 2200.748113] Key type .llcrypt registered [ 2200.883657] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2202.088616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.152183] alg: No test for adler32 (adler32-zlib) [ 2203.331286] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2203.639775] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2205.295257] Key type lgssc registered [ 2206.838910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2214.755222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2214.781342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.230295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.492767] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2226.666845] LustreError: 75084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.684782] LustreError: 75084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2228.696339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.144895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.244702] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2242.403763] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2248.219109] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2254.523584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.842396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.722408] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2262.001102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2267.516869] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2277.122495] Lustre: Failing over lustre-MDT0000 [ 2277.419915] Lustre: server umount lustre-MDT0000 complete [ 2278.370055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.374434] LustreError: 75081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.395488] Lustre: Skipped 1 previous similar message [ 2278.435244] LustreError: 75081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2280.320745] LustreError: 75081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2280.360124] LustreError: 75081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2283.490760] LustreError: 75084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.520859] LustreError: 75084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2286.833307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.984333] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.186425] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2287.310297] LustreError: 78174:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2287.315402] LustreError: 78174:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -9 [ 2287.324425] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2287.324425] [ 2287.336704] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2287.350748] LustreError: 78162:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2287.359115] LustreError: 78162:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2287.366626] LustreError: 78162:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2287.374441] LustreError: 78162:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2287.394983] Lustre: server umount lustre-MDT0000 complete [ 2287.398918] LustreError: 78162:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2288.622806] LustreError: 75085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.650859] LustreError: 75085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2293.630695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.775922] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.185380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.246651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2295.664720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2299.288376] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2299.384335] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2299.425394] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2299.495365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2309.088279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.100246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.116955] Lustre: Skipped 1 previous similar message [ 2309.121937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.624581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2314.464751] Lustre: server umount lustre-OST0000 complete [ 2319.846359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.854522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.854647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2319.858819] Lustre: Skipped 2 previous similar messages [ 2319.890508] Lustre: Skipped 1 previous similar message [ 2324.374735] Lustre: server umount lustre-MDT0000 complete [ 2324.965885] LustreError: 75084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.985803] LustreError: 75084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2327.732484] LustreError: 75064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946776 with bad export cookie 9796647575200707006 [ 2327.734638] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.757311] LustreError: 75064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2328.158589] Lustre: server umount lustre-MDT0001 complete [ 2339.327059] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2342.053555] Key type lgssc unregistered [ 2342.377394] LNet: 79952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.391092] LNetError: 79952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.407435] LNet: Removed LNI 192.168.203.149@tcp [ 2343.257284] Key type .llcrypt unregistered [ 2343.259919] Key type ._llcrypt unregistered [ 2354.754581] Key type ._llcrypt registered [ 2354.760141] Key type .llcrypt registered [ 2354.869910] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2355.154929] Key type .llcrypt unregistered [ 2355.157805] Key type ._llcrypt unregistered [ 2375.146501] Key type ._llcrypt registered [ 2375.151718] Key type .llcrypt registered [ 2375.282726] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2376.343376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.451718] alg: No test for adler32 (adler32-zlib) [ 2377.625936] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2377.913074] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2379.615253] Key type lgssc registered [ 2380.867747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2388.069876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2388.084206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.618361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.021795] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2399.201450] LustreError: 81845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.211918] LustreError: 81845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2402.181569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.562406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2407.308445] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2414.787719] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2420.251874] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2425.973157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2426.313914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.024025] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2434.562513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2438.675182] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2449.894576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.900555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2449.900808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.911148] Lustre: Skipped 1 previous similar message [ 2454.491217] Lustre: server umount lustre-OST0000 complete [ 2459.106081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2459.114242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.125693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.132831] Lustre: Skipped 1 previous similar message [ 2460.150023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2460.152749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.390483] Lustre: server umount lustre-MDT0000 complete [ 2465.252098] LustreError: 82574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.272236] LustreError: 82574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2467.481452] LustreError: 81828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790946916 with bad export cookie 9046930215356892110 [ 2467.489384] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.508802] LustreError: 81828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2467.810741] Lustre: server umount lustre-MDT0001 complete [ 2477.679326] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2480.405766] Key type lgssc unregistered [ 2480.700663] LNet: 85379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.711209] LNetError: 85379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.725576] LNet: Removed LNI 192.168.203.149@tcp [ 2481.548638] Key type .llcrypt unregistered [ 2481.557058] Key type ._llcrypt unregistered [ 2504.098893] Key type ._llcrypt registered [ 2504.103699] Key type .llcrypt registered [ 2504.239751] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2505.154323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2505.294746] alg: No test for adler32 (adler32-zlib) [ 2506.417838] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2506.677501] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2508.351199] Key type lgssc registered [ 2509.610559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2517.264917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2517.295709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.054701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.800592] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2528.236310] LustreError: 86806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.257146] LustreError: 86806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2531.627941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.020362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2536.637096] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2543.837140] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2549.254472] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2555.688684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.963785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.555559] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2563.191519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2568.836526] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2576.555739] Lustre: server umount lustre-OST0000 complete [ 2583.009723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2583.029645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.060898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.032690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.037091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.157723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2589.169339] Lustre: Skipped 2 previous similar messages [ 2590.267914] Lustre: server umount lustre-MDT0000 complete [ 2593.312082] Lustre: 86108:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947026/real 1790947026] req@00000000c70c4451 x1877943994312320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790947042 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2593.346224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.367145] LustreError: 87533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.386565] LustreError: 87533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2593.994418] LustreError: 89587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947042 with bad export cookie 8731661993695778689 [ 2594.004026] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.005291] LustreError: 89587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2594.373042] Lustre: server umount lustre-MDT0001 complete [ 2606.607296] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2609.536794] Key type lgssc unregistered [ 2609.843664] LNet: 90338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.850728] LNetError: 90338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.874988] LNet: Removed LNI 192.168.203.149@tcp [ 2610.999434] Key type .llcrypt unregistered [ 2611.001877] Key type ._llcrypt unregistered [ 2634.173475] Key type ._llcrypt registered [ 2634.176140] Key type .llcrypt registered [ 2634.316753] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2635.622260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2635.673489] alg: No test for adler32 (adler32-zlib) [ 2636.873813] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2637.153754] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2638.847486] Key type lgssc registered [ 2640.536589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.024697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2649.062440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.655484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.897146] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2660.847895] LustreError: 91767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.873278] LustreError: 91767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2665.523679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.823988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.730905] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2678.597856] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2684.212962] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2690.794927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.114403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.224477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2697.781278] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2705.648968] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2724.430736] Lustre: server umount lustre-OST0000 complete [ 2730.976725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.999241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.013152] Lustre: Skipped 1 previous similar message [ 2731.031803] Lustre: Skipped 1 previous similar message [ 2736.097786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2736.107225] Lustre: Skipped 1 previous similar message [ 2739.320322] Lustre: server umount lustre-MDT0000 complete [ 2741.216105] Lustre: 91070:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947174/real 1790947174] req@00000000a8582468 x1877944130630272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790947190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2741.230091] LustreError: 91767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.260736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.298116] LustreError: 91767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2742.246220] Lustre: 91068:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947174/real 1790947174] req@000000003561c24c x1877944130630528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790947190 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2743.082631] LustreError: 91747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947191 with bad export cookie 7353636394055091138 [ 2743.102057] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.614258] Lustre: server umount lustre-MDT0001 complete [ 2754.170561] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2757.592242] Key type lgssc unregistered [ 2757.930384] LNet: 95317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.942961] LNetError: 95317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.979100] LNet: Removed LNI 192.168.203.149@tcp [ 2758.833553] Key type .llcrypt unregistered [ 2758.836266] Key type ._llcrypt unregistered [ 2782.489666] Key type ._llcrypt registered [ 2782.493226] Key type .llcrypt registered [ 2782.623628] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2784.091551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2784.123688] alg: No test for adler32 (adler32-zlib) [ 2785.214591] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 2785.519328] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2787.256141] Key type lgssc registered [ 2788.483811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2797.218325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2797.234373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.833303] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2817.951395] LustreError: 96718:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2844.703390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.340378] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2858.074663] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2864.049555] Lustre: server umount lustre-OST0000 complete [ 2873.601778] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:21:59 (1790947319) [ 2880.310396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.890081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.120926] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2886.116342] LustreError: 98359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.142184] LustreError: 98359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2890.219427] LustreError: 98358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.772607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.144348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2898.458425] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2905.908606] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2911.657976] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2918.813391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.153537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2925.316216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2925.612243] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2933.137391] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2949.089735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.095123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.114641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2949.131264] Lustre: Skipped 1 previous similar message [ 2953.444387] Lustre: server umount lustre-OST0000 complete [ 2959.330504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.349029] Lustre: Skipped 1 previous similar message [ 2959.366124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2959.383958] Lustre: Skipped 1 previous similar message [ 2964.449023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.458346] Lustre: Skipped 1 previous similar message [ 2969.569937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.573642] Lustre: Skipped 1 previous similar message [ 2971.104233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2971.271190] Lustre: server umount lustre-MDT0000 complete [ 2974.690046] LustreError: 98359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.717986] LustreError: 98359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2975.517129] LustreError: 98339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947424 with bad export cookie 186446647627435389 [ 2975.519768] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.540737] LustreError: 98339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2975.981358] Lustre: server umount lustre-MDT0001 complete [ 2987.864927] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2991.329666] Key type lgssc unregistered [ 2991.709462] LNet: 102051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2991.722375] LNetError: 102051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2991.752118] LNet: Removed LNI 192.168.203.149@tcp [ 2992.777155] Key type .llcrypt unregistered [ 2992.778938] Key type ._llcrypt unregistered [ 3037.311659] Key type ._llcrypt registered [ 3037.314678] Key type .llcrypt registered [ 3037.493229] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3038.587407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3038.697071] alg: No test for adler32 (adler32-zlib) [ 3040.034213] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3040.388663] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3042.079513] Key type lgssc registered [ 3043.606774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3052.206613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3052.261295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.767635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.688950] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3063.265939] LustreError: 104250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.294598] LustreError: 104250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3067.974570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.478131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3074.031769] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3082.780616] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3089.032460] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3095.671918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.992056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.066082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3102.747530] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3110.881745] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3123.688935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.692279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3123.693903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.704758] Lustre: Skipped 1 previous similar message [ 3128.610570] Lustre: server umount lustre-OST0000 complete [ 3133.922798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.926993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.943561] Lustre: Skipped 1 previous similar message [ 3133.960191] Lustre: Skipped 2 previous similar messages [ 3139.040188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.044757] Lustre: Skipped 1 previous similar message [ 3144.161763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.173283] Lustre: Skipped 1 previous similar message [ 3147.231242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3147.425361] Lustre: server umount lustre-MDT0000 complete [ 3149.292467] LustreError: 104247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.330032] LustreError: 104247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3151.132654] LustreError: 106831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947599 with bad export cookie 13324005749571893852 [ 3151.142318] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3151.146531] LustreError: 106831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3151.577684] Lustre: server umount lustre-MDT0001 complete [ 3163.425510] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3166.487506] Key type lgssc unregistered [ 3166.764323] LNet: 107786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3166.773902] LNetError: 107786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3166.806149] LNet: Removed LNI 192.168.203.149@tcp [ 3167.883682] Key type .llcrypt unregistered [ 3167.902281] Key type ._llcrypt unregistered [ 3187.018157] Key type ._llcrypt registered [ 3187.022674] Key type .llcrypt registered [ 3187.154313] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3188.514373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3188.533628] alg: No test for adler32 (adler32-zlib) [ 3189.654871] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3189.967380] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3191.647562] Key type lgssc registered [ 3192.873137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3201.622534] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3201.652173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3217.247421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3222.371227] LustreError: 109043:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 3249.120087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.585273] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3264.252220] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3271.692361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.022559] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3272.029576] LustreError: 110069:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3272.036558] LustreError: 110069:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3272.045042] LustreError: 110069:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3272.139434] Lustre: server umount lustre-MDT0000 complete [ 3272.147927] LustreError: 110069:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3293.151167] Lustre: 108371:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790947725/real 1790947725] req@000000000c45f756 x1877944710466944/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790947741 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3293.205606] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3293.221924] LustreError: 109055:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3313.777979] Lustre: server umount lustre-OST0000 complete [ 3324.840274] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3336.519883] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3377.614921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.911719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3377.932344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3377.990727] Lustre: lustre-MDT0000: new disk, initializing [ 3378.088946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.106224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3382.996175] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3394.328384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.474623] Lustre: 112865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3394.562787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3394.574599] Lustre: Skipped 1 previous similar message [ 3394.581733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3394.687115] Lustre: lustre-MDT0001: new disk, initializing [ 3394.802684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.857087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3394.874183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3400.481237] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3410.628605] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3417.197749] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3425.157672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3425.581200] Lustre: lustre-OST0000: new disk, initializing [ 3425.591834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3425.614460] Lustre: 114831:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3425.719842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.645304] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3435.607739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3435.621458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3435.722240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3444.182258] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3449.856287] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3450.056449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3455.003776] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3455.264838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3461.603911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3461.607067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3461.616896] LustreError: Skipped 1 previous similar message [ 3461.619032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3461.647842] Lustre: Skipped 1 previous similar message [ 3463.396338] Lustre: server umount lustre-OST0000 complete [ 3471.840791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3471.844353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3471.847903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3471.857786] Lustre: Skipped 1 previous similar message [ 3471.868668] Lustre: Skipped 2 previous similar messages [ 3473.521497] Lustre: server umount lustre-MDT0000 complete [ 3477.224231] LustreError: 112856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790947926 with bad export cookie 13117764660736843010 [ 3477.228893] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.233970] LustreError: 112856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3477.725637] Lustre: server umount lustre-MDT0001 complete [ 3486.507961] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:32:12 (1790947932) [ 3489.772530] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3506.391966] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3516.847211] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3555.251822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.738231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3555.820143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3555.977656] Lustre: lustre-MDT0000: new disk, initializing [ 3556.151905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.180688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3561.614381] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3574.579264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.672362] Lustre: 119744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3574.707302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3574.710868] Lustre: Skipped 1 previous similar message [ 3574.769974] Lustre: lustre-MDT0001: new disk, initializing [ 3574.851395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3574.884923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3574.900561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3580.171849] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3589.921665] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3596.768411] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3603.989341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.301808] Lustre: lustre-OST0000: new disk, initializing [ 3604.305836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3604.311871] Lustre: 121712:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3604.458479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.800462] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3612.773182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3612.786049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3612.885632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3621.895114] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3627.514106] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3627.748191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3632.458909] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3632.750944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3638.757431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3638.758586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.768483] LustreError: Skipped 1 previous similar message [ 3638.771634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.789027] Lustre: Skipped 1 previous similar message [ 3638.813808] Lustre: Skipped 1 previous similar message [ 3640.988127] Lustre: server umount lustre-OST0000 complete [ 3646.433028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3646.447638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.467861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3651.128954] Lustre: server umount lustre-MDT0000 complete [ 3654.114454] LustreError: 119751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.132201] LustreError: 119751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3656.075870] LustreError: 119734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948104 with bad export cookie 13117764660736843892 [ 3656.088231] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.094805] LustreError: 119734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3656.470733] Lustre: server umount lustre-MDT0001 complete [ 3667.346775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.934961] LustreError: 123930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.102363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.227787] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3673.573701] LustreError: 123931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.605443] LustreError: 123931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3677.676570] LustreError: 123930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.805209] LustreError: 123931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.929350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.885743] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3700.000776] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3704.982925] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3711.935977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3720.344484] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3729.211743] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3736.644211] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3744.231132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.243921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3744.245714] Lustre: Skipped 1 previous similar message [ 3744.246542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.246546] Lustre: Skipped 2 previous similar messages [ 3744.282600] LustreError: Skipped 1 previous similar message [ 3748.377194] Lustre: server umount lustre-OST0000 complete [ 3754.466039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.486155] Lustre: Skipped 1 previous similar message [ 3764.705441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.708714] Lustre: Skipped 6 previous similar messages [ 3767.263116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3767.477583] Lustre: server umount lustre-MDT0000 complete [ 3769.829270] LustreError: 124659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3769.846982] LustreError: 124659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3770.547667] LustreError: 123910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948219 with bad export cookie 13117764660736844774 [ 3770.560177] LustreError: 123910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3770.565557] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.161348] Lustre: server umount lustre-MDT0001 complete [ 3786.380427] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3789.788597] Key type lgssc unregistered [ 3790.160940] LNet: 127562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3790.169899] LNetError: 127562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3791.207512] LNet: Removed LNI 192.168.203.149@tcp [ 3792.354115] Key type .llcrypt unregistered [ 3792.360900] Key type ._llcrypt unregistered [ 3813.601281] Key type ._llcrypt registered [ 3813.606179] Key type .llcrypt registered [ 3813.685370] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3830.766354] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3831.538077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3831.603659] alg: No test for adler32 (adler32-zlib) [ 3832.749711] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 3832.979893] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3834.679286] Key type lgssc registered [ 3836.397771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3869.763801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3869.807278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.210726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3871.272207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3871.411731] Lustre: lustre-MDT0000: new disk, initializing [ 3871.618190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.646302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3876.380149] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3887.016946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.139937] Lustre: 130733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3887.221697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3887.225375] Lustre: Skipped 1 previous similar message [ 3887.306461] Lustre: lustre-MDT0001: new disk, initializing [ 3887.443669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3887.480154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3887.499566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3891.376085] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3900.512596] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3906.495541] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3913.433826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3913.866634] Lustre: lustre-OST0000: new disk, initializing [ 3913.877793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3913.888844] Lustre: 132702:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3914.034972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.495712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3919.509617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3919.705129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3920.425615] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3932.916773] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3937.813057] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3938.076521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3942.257681] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3942.477660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3945.442803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3945.447229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.454178] LustreError: Skipped 1 previous similar message [ 3945.456139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.504512] Lustre: Skipped 1 previous similar message [ 3950.593934] Lustre: server umount lustre-OST0000 complete [ 3954.152195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3954.162245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.178484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3954.185128] Lustre: Skipped 1 previous similar message [ 3955.681297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3955.682138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.943460] Lustre: server umount lustre-MDT0000 complete [ 3963.072718] LustreError: 130725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948411 with bad export cookie 6186749382485523638 [ 3963.075026] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.082744] LustreError: 130725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3963.404529] Lustre: server umount lustre-MDT0001 complete [ 3971.922946] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:40:17 (1790948417) [ 3978.617511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.005440] LustreError: 135115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.029473] LustreError: 135115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3979.089313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.485263] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3984.357482] LustreError: 135116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.486172] LustreError: 135115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.268238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.580046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.147878] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4004.351686] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4010.088352] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4014.048758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.070835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.077192] Lustre: Skipped 1 previous similar message [ 4018.908493] Lustre: server umount lustre-MDT0000 complete [ 4021.218644] LustreError: 135116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.240388] LustreError: 135116:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4023.167975] LustreError: 135095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948472 with bad export cookie 6186749382485524520 [ 4023.191808] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.197981] LustreError: 135095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4023.533686] Lustre: server umount lustre-MDT0001 complete [ 4031.415622] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:41:17 (1790948477) [ 4038.041653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4053.727677] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4058.847571] LustreError: 137529:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 4084.639843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.315320] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4099.142922] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4101.223436] Lustre: server umount lustre-OST0000 complete [ 4109.135366] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:42:35 (1790948555) [ 4115.372839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.865675] LustreError: 138975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.971453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.347880] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4126.179880] LustreError: 138975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.201556] LustreError: 138975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4128.945208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.517275] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4141.366509] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4146.754726] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4153.025810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.433522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4153.449441] Lustre: Skipped 1 previous similar message [ 4159.983068] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4168.020410] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4175.329095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4175.335322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.350572] Lustre: Skipped 1 previous similar message [ 4175.357333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4175.362166] Lustre: Skipped 2 previous similar messages [ 4181.504992] Lustre: server umount lustre-MDT0000 complete [ 4181.992710] LustreError: 141236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.992710] LustreError: 138970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.027446] LustreError: 138970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4185.554770] LustreError: 139711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948634 with bad export cookie 6186749382485525185 [ 4185.557145] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.570383] LustreError: 139711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4185.949311] Lustre: server umount lustre-MDT0001 complete [ 4199.403794] Lustre: server umount lustre-OST0000 complete [ 4207.217056] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:44:13 (1790948653) [ 4213.528488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.647596] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4227.224315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.703888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4227.709359] Lustre: Skipped 1 previous similar message [ 4232.209807] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4238.058648] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4243.664570] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4250.874255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.296893] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4258.352455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4266.509725] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4270.828760] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4271.015634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4275.113411] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4275.385857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4278.755950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4278.765312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.779362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4278.788975] Lustre: Skipped 3 previous similar messages [ 4278.801072] Lustre: Skipped 4 previous similar messages [ 4284.313776] Lustre: server umount lustre-OST0000 complete [ 4294.137876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4299.233097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4299.240302] Lustre: Skipped 5 previous similar messages [ 4303.327331] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4303.535700] Lustre: server umount lustre-MDT0000 complete [ 4304.352160] LustreError: 142633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.366803] LustreError: 142633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4307.329937] LustreError: 142614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948756 with bad export cookie 6186749382485526256 [ 4307.349285] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.357246] LustreError: 142614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4307.764326] Lustre: server umount lustre-MDT0001 complete [ 4316.937782] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:46:02 (1790948762) [ 4324.859245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4340.511629] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4345.644185] LustreError: 146794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 4371.295550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4371.306766] Lustre: Skipped 1 previous similar message [ 4379.504413] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4387.578154] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4394.056165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.694448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4399.887705] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4409.469718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.069670] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4423.365812] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4429.171903] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4434.503942] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4434.774086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4439.805594] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4440.050587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4445.670342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.694797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4445.697368] Lustre: Skipped 3 previous similar messages [ 4445.698733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.721984] Lustre: Skipped 2 previous similar messages [ 4448.601598] Lustre: server umount lustre-MDT0000 complete [ 4450.793280] LustreError: 147833:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.807854] LustreError: 147833:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4453.080247] LustreError: 146802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790948901 with bad export cookie 6186749382485527278 [ 4453.087555] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.094442] LustreError: 146802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4453.571376] Lustre: server umount lustre-MDT0001 complete [ 4467.760168] Lustre: server umount lustre-OST0000 complete [ 4476.787304] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:48:43 (1790948923) [ 4483.644491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4499.424509] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4504.543518] LustreError: 150991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 4537.620121] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4545.251521] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4552.628578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.333136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4558.436402] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4567.880412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.039723] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4580.765121] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4586.945328] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4594.265073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.493233] Lustre: lustre-OST0001: new disk, initializing [ 4594.496631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4594.504312] Lustre: 153863:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4601.400629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4601.418600] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4601.577728] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4601.640803] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4611.117546] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4615.745679] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4615.947339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4620.241109] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4620.443329] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4622.311883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.318898] Lustre: Skipped 1 previous similar message [ 4622.322417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4628.228273] Lustre: server umount lustre-OST0000 complete [ 4632.553734] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.593551] LustreError: Skipped 1 previous similar message [ 4638.573742] Lustre: server umount lustre-OST0001 complete [ 4648.866396] Lustre: server umount lustre-MDT0000 complete [ 4652.962980] LustreError: 150998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949101 with bad export cookie 6186749382485528230 [ 4652.974040] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4652.993883] LustreError: 150998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4653.441680] Lustre: server umount lustre-MDT0001 complete [ 4678.904168] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:52:05 (1790949125) [ 4680.701376] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4682.504573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4684.754062] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:52:10 (1790949130) [ 4692.065947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4692.365848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4692.391040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4692.621030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.630500] Lustre: Skipped 6 previous similar messages [ 4697.249280] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4706.161932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.262173] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4706.297756] Lustre: 157426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4711.732538] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4719.861463] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4725.378180] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4732.020577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4732.180992] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4738.204477] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4739.692192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4745.897335] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4752.142970] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4752.379576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4757.124774] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4757.358482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4760.032307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4760.039104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.051956] Lustre: Skipped 5 previous similar messages [ 4760.059548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4760.077123] Lustre: Skipped 11 previous similar messages [ 4765.871719] Lustre: server umount lustre-OST0000 complete [ 4789.867456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4795.883065] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.897654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.902713] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4795.917253] Lustre: Skipped 1 previous similar message [ 4796.855329] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4803.895293] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4811.443278] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4811.772143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4817.797537] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4818.172273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4824.996805] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93fbc8dea000.ost_server_uuid 50 [ 4826.696726] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff93fbc8dea000.ost_server_uuid in IDLE state after 0 sec [ 4826.974607] LustreError: 157433:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.49@tcp) returned error from blocking AST (req@000000008f5e3f5d x1877945384960384 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000cd43464c/0x55dbbfc8d96ad800 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0xd43b03ae094d054f expref: 5 pid: 157433 timeout: 4926 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4827.047092] LustreError: lustre-MDT0000: A client on nid 192.168.203.49@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4827.062141] LustreError: 157421:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.203.49@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000cd43464c/0x55dbbfc8d96ad800 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0xd43b03ae094d054f expref: 6 pid: 157433 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4832.751264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4832.758740] LustreError: Skipped 1 previous similar message [ 4838.219682] Lustre: server umount lustre-OST0000 complete [ 4850.980563] Lustre: server umount lustre-MDT0000 complete [ 4852.197726] LustreError: 157433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4852.233136] LustreError: 157433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 4855.234073] LustreError: 157416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949304 with bad export cookie 6186749382485529413 [ 4855.237490] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.250285] LustreError: 157416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4861.890619] Lustre: server umount lustre-MDT0001 complete [ 4873.567212] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4876.521837] Key type lgssc unregistered [ 4876.778573] LNet: 163546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4876.783316] LNetError: 163546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4876.795415] LNet: Removed LNI 192.168.203.149@tcp [ 4877.472732] Key type .llcrypt unregistered [ 4877.475157] Key type ._llcrypt unregistered [ 4905.470549] Key type ._llcrypt registered [ 4905.482197] Key type .llcrypt registered [ 4905.702962] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4907.064680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4907.099953] alg: No test for adler32 (adler32-zlib) [ 4908.303549] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 4908.615826] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4910.343252] Key type lgssc registered [ 4911.573588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4919.975867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4920.001123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4921.842482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.516458] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4932.065960] LustreError: 164973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.090062] LustreError: 164973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4937.088877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4937.202648] LustreError: 164974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4937.620954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4943.032933] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4950.775134] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4956.763600] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4964.835802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4965.212478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.281772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4972.334292] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4980.691606] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4989.261844] Lustre: Failing over lustre-MDT0000 [ 4989.651984] Lustre: server umount lustre-MDT0000 complete [ 4992.997284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4993.003928] LustreError: 164973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.017878] Lustre: Skipped 2 previous similar messages [ 4993.046809] LustreError: 164973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4998.114704] LustreError: 164974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.151659] LustreError: 164974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5002.425768] LustreError: 164955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.49@tcp arrived at 1790949451 with bad export cookie 9299739987681712901 [ 5002.440523] LustreError: 164955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5003.242735] LustreError: 164974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.274936] LustreError: 164974:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5008.354625] Lustre: 164276:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790949441/real 1790949441] req@00000000e9974eb8 x1877946512994816/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790949457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5008.395677] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5010.986619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.474603] LustreError: 164970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.502142] LustreError: 164970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5018.594910] LustreError: 164273:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000790c458f x1877946512999424/t0(0) o250->MGC192.168.203.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5018.957483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.039308] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5024.236531] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5024.254874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5024.362251] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5026.797275] Lustre: lustre-MDT0000: Denying connection for new client fbb37587-bb1e-4f1b-8968-e81eb9a5d3fc (at 192.168.203.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5039.072275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5039.077446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.121106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.714945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.728621] Lustre: Skipped 2 previous similar messages [ 5049.314495] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5049.633739] Lustre: server umount lustre-OST0000 complete [ 5054.178436] LustreError: 168707:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5054.185084] Lustre: 168054:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5054.191782] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5054.207222] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5054.220078] LustreError: 168054:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000072b536ee x1877946513015808/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5054.240376] LustreError: 168054:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5054.251071] LustreError: 168054:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5054.259840] Lustre: lustre-MDT0000: Recovery over after 0:30, of 2 clients 0 recovered and 2 were evicted. [ 5054.615269] Lustre: server umount lustre-MDT0000 complete [ 5054.944774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.973219] Lustre: Skipped 1 previous similar message [ 5054.983610] LustreError: 164970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.009863] LustreError: 164970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5059.096727] LustreError: 167670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949507 with bad export cookie 9299739987681713006 [ 5059.106560] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5059.109095] LustreError: 164273:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5059.111023] LustreError: 167670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5061.585581] Lustre: server umount lustre-MDT0001 complete [ 5073.848945] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5077.241738] Key type lgssc unregistered [ 5077.602742] LNet: 169459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5077.613557] LNetError: 169459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5078.631853] LNet: Removed LNI 192.168.203.149@tcp [ 5079.779115] Key type .llcrypt unregistered [ 5079.784920] Key type ._llcrypt unregistered [ 5106.316814] Key type ._llcrypt registered [ 5106.326122] Key type .llcrypt registered [ 5106.515889] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5108.003714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5108.248917] alg: No test for adler32 (adler32-zlib) [ 5109.302767] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5109.639112] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5111.391841] Key type lgssc registered [ 5112.903108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5120.946808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5120.969613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.516744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.480635] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5132.771389] LustreError: 170902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.788383] LustreError: 170902:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5135.960438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.263235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.919411] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5149.925378] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5157.041609] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5165.022760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5165.517446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.617046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5171.923852] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5179.550269] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5187.040043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5187.041830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.049853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5187.055909] Lustre: Skipped 1 previous similar message [ 5191.622914] Lustre: server umount lustre-OST0000 complete [ 5197.282501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.328244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5197.337053] Lustre: Skipped 1 previous similar message [ 5202.421117] LustreError: 171630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.447529] LustreError: 171630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5202.815080] Lustre: server umount lustre-MDT0000 complete [ 5207.524635] LustreError: 170901:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.539859] LustreError: 170901:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5207.732438] LustreError: 170883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949656 with bad export cookie 8318767381275015146 [ 5207.734317] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5207.756438] LustreError: 170883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5208.016916] Lustre: server umount lustre-MDT0001 complete [ 5219.435951] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5222.659345] Key type lgssc unregistered [ 5223.089812] LNet: 174444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5223.103644] LNetError: 174444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5223.117893] LNet: Removed LNI 192.168.203.149@tcp [ 5223.970995] Key type .llcrypt unregistered [ 5223.973278] Key type ._llcrypt unregistered [ 5255.352476] Key type ._llcrypt registered [ 5255.354933] Key type .llcrypt registered [ 5255.435960] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5256.229572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5256.371610] alg: No test for adler32 (adler32-zlib) [ 5257.422504] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5257.618833] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5259.279249] Key type lgssc registered [ 5260.590892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5269.128082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5269.191868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.965363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.261679] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5280.226583] LustreError: 176163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.238200] LustreError: 176163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5285.348954] LustreError: 176164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.413382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5285.932686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5290.594437] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5298.252779] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5304.299923] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5310.869315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5311.295312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.395592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5317.364265] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5325.304696] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5338.089310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5338.104783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5338.118791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5341.152569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5341.154445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5341.166318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.184685] Lustre: Skipped 1 previous similar message [ 5344.052437] Lustre: server umount lustre-OST0000 complete [ 5351.392470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.401801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5354.575849] Lustre: server umount lustre-MDT0000 complete [ 5356.513913] LustreError: 177166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.551954] LustreError: 177166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5358.845832] LustreError: 176143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949807 with bad export cookie 18171596000999363675 [ 5358.859493] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5358.861965] LustreError: 176143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5359.449560] Lustre: server umount lustre-MDT0001 complete [ 5370.912885] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5374.008941] Key type lgssc unregistered [ 5374.306796] LNet: 179716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5374.317517] LNetError: 179716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5374.336820] LNet: Removed LNI 192.168.203.149@tcp [ 5375.189254] Key type .llcrypt unregistered [ 5375.193064] Key type ._llcrypt unregistered [ 5397.952341] Key type ._llcrypt registered [ 5397.954350] Key type .llcrypt registered [ 5398.083866] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5399.287042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5399.357153] alg: No test for adler32 (adler32-zlib) [ 5400.454491] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5400.758728] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5402.463195] Key type lgssc registered [ 5403.878149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5414.907591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5414.939519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.284133] Lustre: *** cfs_fail_loc=135, val=0*** [ 5416.342177] LustreError: 181253:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5416.347163] LustreError: 181253:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 5416.353917] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5416.353917] [ 5416.360527] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5416.373098] LustreError: 181214:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5416.378910] LustreError: 181214:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5416.386254] LustreError: 181214:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5416.390176] LustreError: 181214:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5416.431918] Lustre: server umount lustre-MDT0000 complete [ 5416.433822] LustreError: 181214:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5422.932075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.485272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.817847] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5433.827622] LustreError: 181625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.848455] LustreError: 181625:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5437.836782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.167284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5442.744941] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5450.843537] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5456.485600] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5458.912708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.915120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5458.932720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.016341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.025737] Lustre: Skipped 1 previous similar message [ 5468.128450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5468.140651] Lustre: Skipped 1 previous similar message [ 5473.247335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5473.251374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5473.284355] Lustre: Skipped 1 previous similar message [ 5473.508596] Lustre: server umount lustre-MDT0000 complete [ 5477.257609] LustreError: 181604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790949926 with bad export cookie 11905027101827272829 [ 5477.266699] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5477.274348] LustreError: 181604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5477.924673] Lustre: server umount lustre-MDT0001 complete [ 5488.873336] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5492.043192] Key type lgssc unregistered [ 5492.372672] LNet: 183947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5492.381914] LNetError: 183947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5492.407740] LNet: Removed LNI 192.168.203.149@tcp [ 5493.323332] Key type .llcrypt unregistered [ 5493.327095] Key type ._llcrypt unregistered [ 5515.293590] Key type ._llcrypt registered [ 5515.297817] Key type .llcrypt registered [ 5515.444727] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5515.719614] Key type .llcrypt unregistered [ 5515.723659] Key type ._llcrypt unregistered [ 5534.313754] Key type ._llcrypt registered [ 5534.315930] Key type .llcrypt registered [ 5534.473593] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5535.281994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5535.357413] alg: No test for adler32 (adler32-zlib) [ 5536.398741] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5536.661638] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5538.319307] Key type lgssc registered [ 5539.261664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5547.463237] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5547.528988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5563.231503] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5568.351427] LustreError: 186095:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 5595.103558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.927110] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5609.879121] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5616.409580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.990136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5617.054354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5622.027622] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5622.245279] LustreError: 186120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.279356] LustreError: 186120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5627.363914] LustreError: 186121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.417194] LustreError: 187132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.455695] LustreError: 187132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5629.753823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.175869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5634.195298] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5641.279631] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.795595] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5649.621562] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5658.082256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5658.092546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.109419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.155558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.159845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5661.183618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.842724] Lustre: server umount lustre-OST0000 complete [ 5671.392841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.394415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.401808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.419651] Lustre: Skipped 1 previous similar message [ 5671.426862] Lustre: Skipped 1 previous similar message [ 5673.778530] Lustre: server umount lustre-MDT0000 complete [ 5676.514034] LustreError: 187137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5676.535743] LustreError: 187137:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5677.370952] LustreError: 188760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950126 with bad export cookie 13540585647389048899 [ 5677.372206] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5677.383481] LustreError: 188760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5677.906283] Lustre: server umount lustre-MDT0001 complete [ 5687.764752] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5690.880743] Key type lgssc unregistered [ 5691.213400] LNet: 190012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5691.219390] LNetError: 190012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5692.263363] LNet: Removed LNI 192.168.203.149@tcp [ 5693.102497] Key type .llcrypt unregistered [ 5693.106859] Key type ._llcrypt unregistered [ 5714.193738] Key type ._llcrypt registered [ 5714.195844] Key type .llcrypt registered [ 5714.308298] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5715.227936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5715.274234] alg: No test for adler32 (adler32-zlib) [ 5716.349645] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5716.608907] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5718.303323] Key type lgssc registered [ 5719.627761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5727.253226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5727.282938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.780436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.071758] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5738.978478] LustreError: 191421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5738.994928] LustreError: 191421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5740.660381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.015826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5745.432064] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5752.172182] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5757.358352] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5764.478848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5764.889731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.965992] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5774.135846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5780.142984] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5785.915614] Lustre: Failing over lustre-MDT0000 [ 5786.323973] Lustre: server umount lustre-MDT0000 complete [ 5787.105722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5787.114697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.128895] LustreError: 192148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.558346] LustreError: 191416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.670860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.709094] Lustre: Skipped 1 previous similar message [ 5793.643787] LustreError: 191416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.694365] LustreError: 191416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5798.756751] LustreError: 193308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.768613] LustreError: 193308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5804.828612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.911898] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5804.968533] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.49@tcp (not set up) [ 5805.216600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.266568] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5808.991561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5809.762834] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5810.672776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5810.714272] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5810.789073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5814.250831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5821.993173] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5831.145308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.164221] Lustre: Skipped 1 previous similar message [ 5831.164677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.187647] Lustre: Skipped 4 previous similar messages [ 5832.631395] Lustre: server umount lustre-OST0000 complete [ 5841.376369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5841.376541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.388069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5841.408509] Lustre: Skipped 1 previous similar message [ 5842.855982] Lustre: server umount lustre-MDT0000 complete [ 5846.503786] LustreError: 191415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.532349] LustreError: 191415:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5846.787904] LustreError: 191400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950295 with bad export cookie 15422480484179627442 [ 5846.788924] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5846.795855] LustreError: 191400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5847.324803] Lustre: server umount lustre-MDT0001 complete [ 5858.453352] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5861.260485] Key type lgssc unregistered [ 5861.544837] LNet: 196305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5861.552758] LNetError: 196305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5861.572322] LNet: Removed LNI 192.168.203.149@tcp [ 5862.362939] Key type .llcrypt unregistered [ 5862.365346] Key type ._llcrypt unregistered [ 5884.744100] Key type ._llcrypt registered [ 5884.745635] Key type .llcrypt registered [ 5884.855941] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5885.694158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5885.718278] alg: No test for adler32 (adler32-zlib) [ 5886.839365] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 5887.034124] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5888.679566] Key type lgssc registered [ 5889.749322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5896.916203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5896.933340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.383771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.792468] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5908.961568] LustreError: 197735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.976799] LustreError: 197735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5911.049959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.523104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5915.423915] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5923.748762] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.319639] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5935.516169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5935.931497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.450530] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5944.179871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5948.806922] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5951.079485] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5960.194412] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5972.918351] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5980.140355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5980.147531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.152737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5980.161729] Lustre: Skipped 1 previous similar message [ 5984.969047] Lustre: server umount lustre-OST0000 complete [ 5990.369419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.386529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5990.394040] Lustre: Skipped 2 previous similar messages [ 5994.861522] Lustre: server umount lustre-MDT0000 complete [ 5995.489912] LustreError: 198783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.507755] LustreError: 198783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5998.455236] LustreError: 197714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950447 with bad export cookie 1708458275973171267 [ 5998.463693] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5998.471928] LustreError: 197714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5998.972129] Lustre: server umount lustre-MDT0001 complete [ 6011.576933] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6014.601955] Key type lgssc unregistered [ 6014.913072] LNet: 201586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6014.926270] LNetError: 201586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6014.949253] LNet: Removed LNI 192.168.203.149@tcp [ 6015.732207] Key type .llcrypt unregistered [ 6015.740113] Key type ._llcrypt unregistered [ 6035.409928] Key type ._llcrypt registered [ 6035.413710] Key type .llcrypt registered [ 6035.528573] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6036.511356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6036.631308] alg: No test for adler32 (adler32-zlib) [ 6037.730926] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6038.050050] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6039.751204] Key type lgssc registered [ 6040.814890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.683568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6047.718772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.092365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6053.321441] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6058.467508] LustreError: 202869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.485533] LustreError: 202869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6061.762966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.287301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.880343] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6073.606570] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6078.979274] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6084.769387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6085.129239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6090.448835] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6093.358914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6097.153810] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6102.114692] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6107.196921] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6116.912127] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6123.871532] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6132.192564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6132.210554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.244627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6134.240585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6134.243608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6134.249864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6134.284805] Lustre: Skipped 1 previous similar message [ 6135.975703] Lustre: server umount lustre-OST0000 complete [ 6143.968568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6143.980384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.999576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6145.958282] Lustre: server umount lustre-MDT0000 complete [ 6149.460332] LustreError: 203602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950598 with bad export cookie 597277658695227010 [ 6149.471280] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.476042] LustreError: 203602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6149.811039] Lustre: server umount lustre-MDT0001 complete [ 6160.032354] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6162.760709] Key type lgssc unregistered [ 6163.062950] LNet: 207645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6163.073655] LNetError: 207645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6163.093635] LNet: Removed LNI 192.168.203.149@tcp [ 6164.077774] Key type .llcrypt unregistered [ 6164.081076] Key type ._llcrypt unregistered [ 6187.607737] Key type ._llcrypt registered [ 6187.610498] Key type .llcrypt registered [ 6187.727426] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6188.785846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6188.985375] alg: No test for adler32 (adler32-zlib) [ 6190.111824] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6190.372669] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6192.055158] Key type lgssc registered [ 6193.374880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6199.825662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6199.852888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.424699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.001198] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6211.561391] LustreError: 209075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.589230] LustreError: 209075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6213.905780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.227965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6218.781347] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6225.542473] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.671629] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6236.841728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6237.161996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.068684] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6246.404537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6248.961260] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.778312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.784047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.803685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6266.860288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.862689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6266.863452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6268.184707] Lustre: server umount lustre-OST0000 complete [ 6271.968986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.982324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6271.985109] Lustre: Skipped 1 previous similar message [ 6277.097740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6277.109856] Lustre: Skipped 1 previous similar message [ 6277.836796] Lustre: server umount lustre-MDT0000 complete [ 6280.788171] LustreError: 209056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950729 with bad export cookie 10080587552493619749 [ 6280.788745] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6280.798826] LustreError: 209056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6281.131633] Lustre: server umount lustre-MDT0001 complete [ 6291.455397] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6294.011332] Key type lgssc unregistered [ 6294.306633] LNet: 212755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6294.312152] LNetError: 212755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6295.338949] LNet: Removed LNI 192.168.203.149@tcp [ 6296.175678] Key type .llcrypt unregistered [ 6296.181127] Key type ._llcrypt unregistered [ 6314.461798] Key type ._llcrypt registered [ 6314.465256] Key type .llcrypt registered [ 6314.610311] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6315.524311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6315.661849] alg: No test for adler32 (adler32-zlib) [ 6316.733143] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6316.981986] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6318.655196] Key type lgssc registered [ 6319.569047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6326.423523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6326.441937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.856776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6331.958686] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6338.019625] LustreError: 214055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.050307] LustreError: 214055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6340.272368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.656162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6344.538581] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6351.060261] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6356.503927] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.608954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.036948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.859884] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6372.286521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6375.782687] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6391.790549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.807613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.817047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.803551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.805158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6392.818504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.808806] Lustre: server umount lustre-OST0000 complete [ 6402.016513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6402.028284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.047463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.055138] Lustre: Skipped 1 previous similar message [ 6405.625437] Lustre: server umount lustre-MDT0000 complete [ 6408.160316] LustreError: 214051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6408.183597] LustreError: 214051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6408.860465] LustreError: 215901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790950857 with bad export cookie 9181295276805300449 [ 6408.865640] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.867054] LustreError: 215901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6409.167959] Lustre: server umount lustre-MDT0001 complete [ 6418.863910] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6422.531294] Key type lgssc unregistered [ 6422.996797] LNet: 217738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6423.006599] LNetError: 217738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6423.023836] LNet: Removed LNI 192.168.203.149@tcp [ 6423.882616] Key type .llcrypt unregistered [ 6423.884473] Key type ._llcrypt unregistered [ 6443.298317] Key type ._llcrypt registered [ 6443.302543] Key type .llcrypt registered [ 6443.404195] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6443.965756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6444.011094] alg: No test for adler32 (adler32-zlib) [ 6445.109145] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6445.419720] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6447.151493] Key type lgssc registered [ 6448.390663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6455.342099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6455.380344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6456.655285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.001277] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6467.042663] LustreError: 219020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6467.068506] LustreError: 219020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6469.795556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.204978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6474.866639] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6481.854241] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6486.617080] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.515963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.950260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.372825] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6502.195561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6504.743599] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6513.464488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.559502] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6513.780589] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.907555] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6524.572750] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.896899] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6537.529398] Lustre: Permanently deactivating lustre-OST0001 [ 6537.542540] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6545.980810] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6552.019285] Lustre: Permanently reactivating lustre-OST0001 [ 6552.058982] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6552.067918] Lustre: Skipped 2 previous similar messages [ 6561.842203] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.858461] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6563.141384] Lustre: server umount lustre-OST0001 complete [ 6566.884023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6566.884421] LustreError: 220986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6566.889387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6566.899979] Lustre: Skipped 1 previous similar message [ 6572.009433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6572.009746] LustreError: 220842:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6572.017464] Lustre: Skipped 2 previous similar messages [ 6572.870981] Lustre: server umount lustre-OST0000 complete [ 6577.632620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6577.641439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.655390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.193124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6582.239151] LustreError: 222901:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 6585.314473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6585.319768] Lustre: Skipped 3 previous similar messages [ 6585.401966] Lustre: server umount lustre-MDT0000 complete [ 6588.297054] LustreError: 218999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951037 with bad export cookie 5650524666706252729 [ 6588.306387] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.314035] LustreError: 218999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6588.595789] Lustre: server umount lustre-MDT0001 complete [ 6597.766828] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6600.044411] Key type lgssc unregistered [ 6600.337732] LNet: 224054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6600.346058] LNetError: 224054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6600.362699] LNet: Removed LNI 192.168.203.149@tcp [ 6601.219944] Key type .llcrypt unregistered [ 6601.221860] Key type ._llcrypt unregistered [ 6633.135501] Key type ._llcrypt registered [ 6633.136880] Key type .llcrypt registered [ 6633.206725] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6634.122214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6634.163950] alg: No test for adler32 (adler32-zlib) [ 6635.228624] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6635.489262] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6637.151245] Key type lgssc registered [ 6638.365108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6645.119406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6645.162181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.388926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6646.423091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6646.541283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6650.905861] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6658.915912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.069037] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6659.116907] Lustre: 226266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6659.416426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.928562] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6670.840836] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6675.968478] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6682.390341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.667865] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6682.899816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6688.534963] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6692.939690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6694.470505] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6696.976166] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6703.413474] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6712.013863] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6721.391948] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6729.494939] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6746.706867] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6746.713262] Lustre: Skipped 1 previous similar message [ 6780.366961] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6780.372264] Lustre: Skipped 3 previous similar messages [ 6844.371511] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6844.383463] Lustre: Skipped 7 previous similar messages [ 6882.279153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.280330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6882.287450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6882.297165] Lustre: Skipped 1 previous similar message [ 6882.303076] LustreError: Skipped 1 previous similar message [ 6884.215593] Lustre: server umount lustre-OST0000 complete [ 6889.956469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6889.973468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.992026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6890.001598] Lustre: Skipped 1 previous similar message [ 6892.522665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6892.526989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.558407] Lustre: Skipped 1 previous similar message [ 6897.633319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6897.648902] Lustre: Skipped 1 previous similar message [ 6902.239164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6902.432357] Lustre: server umount lustre-MDT0000 complete [ 6905.824607] LustreError: 226256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951354 with bad export cookie 14478079347723166714 [ 6905.830816] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6905.843663] LustreError: 226256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6906.192484] Lustre: server umount lustre-MDT0001 complete [ 6916.462461] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6919.189581] Key type lgssc unregistered [ 6919.536888] LNet: 231067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6919.550154] LNetError: 231067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6919.569775] LNet: Removed LNI 192.168.203.149@tcp [ 6920.572178] Key type .llcrypt unregistered [ 6920.579574] Key type ._llcrypt unregistered [ 6947.323641] Key type ._llcrypt registered [ 6947.326421] Key type .llcrypt registered [ 6947.484800] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6948.504163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.626861] alg: No test for adler32 (adler32-zlib) [ 6949.639696] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 6949.842066] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6951.560400] Key type lgssc registered [ 6952.798974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6960.648874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6960.677135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.268190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6966.974453] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6972.402035] LustreError: 232496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.429054] LustreError: 232496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6975.955495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.560456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6982.530453] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6989.645364] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6994.936286] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7001.092842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.470996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.675795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7007.992329] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7014.152978] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.312877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7033.325808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.347270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7033.350058] Lustre: Skipped 1 previous similar message [ 7033.365601] Lustre: Skipped 1 previous similar message [ 7037.143479] Lustre: server umount lustre-OST0000 complete [ 7043.041851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7043.047178] LustreError: Skipped 1 previous similar message [ 7043.053172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.076704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.957147] Lustre: server umount lustre-MDT0000 complete [ 7048.678532] LustreError: 233423:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.707909] LustreError: 233423:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7050.145864] LustreError: 232477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790951498 with bad export cookie 16713574193590082453 [ 7050.146952] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.159566] LustreError: 232477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7050.528395] Lustre: server umount lustre-MDT0001 complete [ 7059.223811] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7061.991536] Key type lgssc unregistered [ 7062.373601] LNet: 236182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7062.392076] LNetError: 236182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7062.429041] LNet: Removed LNI 192.168.203.149@tcp [ 7063.267450] Key type .llcrypt unregistered [ 7063.269398] Key type ._llcrypt unregistered [ 7082.477708] Key type ._llcrypt registered [ 7082.479633] Key type .llcrypt registered [ 7082.575674] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7083.505876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7083.563382] alg: No test for adler32 (adler32-zlib) [ 7084.561659] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7084.745719] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 7086.383458] Key type lgssc registered [ 7087.513790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7359.562652] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7362.452662] Key type lgssc unregistered [ 7362.854703] LNet: 238199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.863117] LNetError: 238199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.888651] LNet: Removed LNI 192.168.203.149@tcp [ 7364.083795] Key type .llcrypt unregistered [ 7364.087131] Key type ._llcrypt unregistered [ 7389.763341] Key type ._llcrypt registered [ 7389.765759] Key type .llcrypt registered [ 7389.908857] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 7412.732862] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7413.623548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7413.645621] alg: No test for adler32 (adler32-zlib) [ 7414.811541] Lustre: Lustre: Build Version: 2.17.59_2_gbfaf0e7 [ 7415.087541] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 7416.855282] Key type lgssc registered [ 7417.968702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7448.652442] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7153 sec ========== 10:38:14 (1790951894) [ 7450.871689] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:38:16 (1790951896) === [ 7454.522617] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:38:20 (1790951900) === [ 7491.968269] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7495.468499] Key type lgssc unregistered [ 7495.803248] LNet: 243451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7495.818410] LNetError: 243451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7495.856591] LNet: Removed LNI 192.168.203.149@tcp [ 7496.875515] Key type .llcrypt unregistered [ 7496.877277] Key type ._llcrypt unregistered