[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 552922340 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006935] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.010014] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013084] *** VALIDATE selinux *** [ 0.022079] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027162] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028207] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030154] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032146] *** VALIDATE tmpfs *** [ 0.033508] *** VALIDATE proc *** [ 0.034282] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043702] debug: unmapping init [mem 0xffffffff87859000-0xffffffff87860fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046942] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048036] ... version: 2 [ 0.049023] ... bit width: 48 [ 0.050026] ... generic registers: 4 [ 0.051024] ... value mask: 0000ffffffffffff [ 0.052023] ... max period: 00007fffffffffff [ 0.053026] ... fixed-purpose events: 3 [ 0.054025] ... event mask: 000000070000000f [ 0.055393] rcu: Hierarchical SRCU implementation. [ 0.057663] smp: Bringing up secondary CPUs ... [ 0.058638] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.070109] smp: Brought up 1 node, 4 CPUs [ 0.072022] smpboot: Max logical packages: 1 [ 0.073014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.210124] node 0 deferred pages initialised in 135ms [ 0.213366] devtmpfs: initialized [ 0.214295] x86/mm: Memory block size: 128MB [ 0.217909] gcov: version magic: 0x41383552 [ 0.220361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.221164] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.222285] pinctrl core: initialized pinctrl subsystem [ 0.223218] [ 0.223728] ************************************************************* [ 0.224018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225016] ** ** [ 0.226015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.227015] ** ** [ 0.228014] ** This means that this kernel is built to expose internal ** [ 0.229016] ** IOMMU data structures, which may compromise security on ** [ 0.230013] ** your system. ** [ 0.231014] ** ** [ 0.232016] ** If you see this message and you are not debugging the ** [ 0.233014] ** kernel, report this immediately to your vendor! ** [ 0.234014] ** ** [ 0.235013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.236011] ************************************************************* [ 0.237793] NET: Registered protocol family 16 [ 0.238537] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.239067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.241455] cpuidle: using governor menu [ 0.243107] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245614] PCI: Using configuration type 1 for base access [ 0.248148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.253369] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.254021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.256033] cryptd: max_cpu_qlen set to 1000 [ 0.258303] ACPI: Added _OSI(Module Device) [ 0.259011] ACPI: Added _OSI(Processor Device) [ 0.260013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.265590] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.268923] ACPI: Interpreter enabled [ 0.269069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.270014] ACPI: Using IOAPIC for interrupt routing [ 0.271137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272470] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.282779] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287472] acpiphp: Slot [2] registered [ 0.288234] acpiphp: Slot [5] registered [ 0.289197] acpiphp: Slot [6] registered [ 0.290195] acpiphp: Slot [7] registered [ 0.291191] acpiphp: Slot [8] registered [ 0.292184] acpiphp: Slot [9] registered [ 0.293231] acpiphp: Slot [10] registered [ 0.294223] acpiphp: Slot [3] registered [ 0.295149] acpiphp: Slot [4] registered [ 0.296143] acpiphp: Slot [11] registered [ 0.297154] acpiphp: Slot [12] registered [ 0.298140] acpiphp: Slot [13] registered [ 0.299109] acpiphp: Slot [14] registered [ 0.300191] acpiphp: Slot [15] registered [ 0.301223] acpiphp: Slot [16] registered [ 0.302188] acpiphp: Slot [17] registered [ 0.303187] acpiphp: Slot [18] registered [ 0.304322] acpiphp: Slot [19] registered [ 0.305118] acpiphp: Slot [20] registered [ 0.306086] acpiphp: Slot [21] registered [ 0.307023] acpiphp: Slot [22] registered [ 0.308132] acpiphp: Slot [23] registered [ 0.309078] acpiphp: Slot [24] registered [ 0.310057] acpiphp: Slot [25] registered [ 0.311034] acpiphp: Slot [26] registered [ 0.312034] acpiphp: Slot [27] registered [ 0.313099] acpiphp: Slot [28] registered [ 0.314153] acpiphp: Slot [29] registered [ 0.315102] acpiphp: Slot [30] registered [ 0.317100] acpiphp: Slot [31] registered [ 0.318087] PCI host bridge to bus 0000:00 [ 0.320023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336133] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340337] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349888] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355576] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.360016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.366316] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368755] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.371044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374892] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401666] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452716] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.525021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.556026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.576010] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.596016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.605026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.621016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.631383] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.638017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.645017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667138] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.682184] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.706022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.719018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.739013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.754804] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.764016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.772017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.793020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.806023] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.808509] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.811439] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.814374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.820224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.829310] iommu: Default domain type: Passthrough [ 0.831391] SCSI subsystem initialized [ 0.833169] ACPI: bus type USB registered [ 0.837154] usbcore: registered new interface driver usbfs [ 0.839083] usbcore: registered new interface driver hub [ 0.841080] usbcore: registered new device driver usb [ 0.843160] pps_core: LinuxPPS API ver. 1 registered [ 0.845014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.848059] PTP clock support registered [ 0.851085] EDAC MC: Ver: 3.0.0 [ 0.853029] PCI: Using ACPI for IRQ routing [ 0.854000] NetLabel: Initializing [ 0.854008] NetLabel: domain hash size = 128 [ 0.855000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.858082] NetLabel: unlabeled traffic allowed by default [ 0.860366] vgaarb: loaded [ 0.862337] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.864015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.870717] clocksource: Switched to clocksource kvm-clock [ 0.993184] VFS: Disk quotas dquot_6.6.0 [ 0.995306] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.997954] *** VALIDATE ramfs *** [ 0.999185] *** VALIDATE hugetlbfs *** [ 1.000775] pnp: PnP ACPI init [ 1.003329] pnp: PnP ACPI: found 6 devices [ 1.022883] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.026468] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.028922] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.031405] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.033953] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.036588] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.039585] NET: Registered protocol family 2 [ 1.042455] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.048228] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.052049] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.057422] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.060948] TCP: Hash tables configured (established 65536 bind 65536) [ 1.063953] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.067116] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069924] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.072957] NET: Registered protocol family 1 [ 1.076710] RPC: Registered named UNIX socket transport module. [ 1.079225] RPC: Registered udp transport module. [ 1.081241] RPC: Registered tcp transport module. [ 1.083292] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.086124] NET: Registered protocol family 44 [ 1.087755] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.090745] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.093512] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.096636] PCI: CLS 0 bytes, default 64 [ 1.098572] Unpacking initramfs... [ 2.852965] debug: unmapping init [mem 0xffff99743cc54000-0xffff99743ffbffff] [ 2.857956] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.860438] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.863729] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.504522] Initialise system trusted keyrings [ 3.507482] Key type blacklist registered [ 3.563000] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.599076] zbud: loaded [ 3.605209] *** VALIDATE nfs *** [ 3.608504] *** VALIDATE nfs4 *** [ 3.613150] pstore: using deflate compression [ 3.623293] Platform Keyring initialized [ 3.749860] NET: Registered protocol family 38 [ 3.752212] Key type asymmetric registered [ 3.754183] Asymmetric key parser 'x509' registered [ 3.756546] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.760343] io scheduler mq-deadline registered [ 3.762363] io scheduler kyber registered [ 3.764122] io scheduler bfq registered [ 3.765697] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.769402] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.773249] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.776840] ACPI: Power Button [PWRF] [ 3.783346] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.792680] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.933240] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.950303] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.120906] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.181847] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.216700] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.222316] Non-volatile memory driver v1.3 [ 4.224277] Linux agpgart interface v0.103 [ 4.257402] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.260609] vda: detected capacity change from 0 to 76787712 [ 4.339230] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.342498] vdb: detected capacity change from 0 to 1073741824 [ 4.367234] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.370954] vdc: detected capacity change from 0 to 2621440000 [ 4.395381] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.399218] vdd: detected capacity change from 0 to 2621440000 [ 4.421509] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.425134] vde: detected capacity change from 0 to 4294967296 [ 4.441889] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.445258] vdf: detected capacity change from 0 to 4294967296 [ 4.452677] libphy: Fixed MDIO Bus: probed [ 4.458494] usbcore: registered new interface driver usbserial_generic [ 4.461407] usbserial: USB Serial support registered for generic [ 4.464062] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.469127] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.471171] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.474193] mousedev: PS/2 mouse device common for all mice [ 4.477993] rtc_cmos 00:05: RTC can wake from S4 [ 4.482118] rtc_cmos 00:05: registered as rtc0 [ 4.485602] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.488591] intel_pstate: CPU model not supported [ 4.492431] hid: raw HID events driver (C) Jiri Kosina [ 4.493862] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.495097] usbcore: registered new interface driver usbhid [ 4.495102] usbhid: USB HID core driver [ 4.495245] drop_monitor: Initializing network drop monitor service [ 4.495377] Initializing XFRM netlink socket [ 4.495836] NET: Registered protocol family 10 [ 4.497110] Segment Routing with IPv6 [ 4.497153] NET: Registered protocol family 17 [ 4.498773] mpls_gso: MPLS GSO support [ 4.507534] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.525152] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.538368] RAS: Correctable Errors collector initialized. [ 4.540739] AVX version of gcm_enc/dec engaged. [ 4.542571] AES CTR mode by8 optimization enabled [ 4.670285] sched_clock: Marking stable (4670221728, 0)->(5706599479, -1036377751) [ 4.673261] registered taskstats version 1 [ 4.675459] Loading compiled-in X.509 certificates [ 4.679279] zswap: loaded using pool lzo/zbud [ 4.708445] Key type big_key registered [ 4.723325] Key type encrypted registered [ 4.724632] ima: No TPM chip found, activating TPM-bypass! [ 4.726774] ima: Allocated hash algorithm: sha1 [ 4.728264] ima: No architecture policies found [ 4.730267] evm: Initialising EVM extended attributes: [ 4.732253] evm: security.selinux [ 4.733846] evm: security.ima [ 4.735278] evm: security.capability [ 4.736919] evm: HMAC attrs: 0x1 [ 4.739813] rtc_cmos 00:05: setting system clock to 2026-09-07 20:27:27 UTC (1788812847) [ 4.747165] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 4.750825] debug: unmapping init [mem 0xffffffff87582000-0xffffffff87858fff] [ 4.758220] Write protecting the kernel read-only data: 28672k [ 4.761587] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 4.765188] debug: unmapping init [mem 0xffffffff86514000-0xffffffff865fffff] [ 4.816955] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.827453] systemd[1]: Detected virtualization kvm. [ 4.829839] systemd[1]: Detected architecture x86-64. [ 4.832295] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.882344] systemd[1]: No hostname configured. [ 4.884044] systemd[1]: Set hostname to . [ 4.886077] random: systemd: uninitialized urandom read (16 bytes read) [ 4.888225] systemd[1]: Initializing machine ID from random generator. [ 5.018094] random: ln: uninitialized urandom read (6 bytes read) [ 5.246089] random: systemd: uninitialized urandom read (16 bytes read) [ 5.249995] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.259396] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.265533] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.518124] device-mapper: uevent: version 1.0.3 [ 6.520584] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 8.273744] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.326709] random: fast init done [ 8.888302] scsi host0: ata_piix [ 8.906939] scsi host1: ata_piix [ 8.930491] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.942600] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.281538] random: crng init done [ 13.288870] random: 7 urandom warning(s) missed due to ratelimiting [ 15.792618] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.811160] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.960782] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.024317] SELinux: Disabled at runtime. [ 21.155824] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.174018] systemd[1]: Detected virtualization kvm. [ 21.178090] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.103489] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.107694] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.133695] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.138156] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.206797] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.218099] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.248062] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ 23.714982] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 25.115907] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.225094] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.353865] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.791564] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.935858] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.802709] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.208096] NFS: Registering the id_resolver key type [ 32.210384] Key type id_resolver registered [ 32.211851] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 74.489224] hrtimer: interrupt took 2322829 ns [ 89.725166] libcfs: loading out-of-tree module taints kernel. [ 89.775276] Key type ._llcrypt registered [ 89.777402] Key type .llcrypt registered [ 89.876665] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 104.685163] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 105.666645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.693394] alg: No test for adler32 (adler32-zlib) [ 106.986512] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 107.658410] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 109.369405] Key type lgssc registered [ 110.588760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.399451] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.228619] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 163.787988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 163.816399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.007938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.027183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.103460] Lustre: lustre-MDT0000: new disk, initializing [ 165.177790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.191975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.506290] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.189699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.279593] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.314150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.319850] Lustre: Skipped 1 previous similar message [ 179.399525] Lustre: lustre-MDT0001: new disk, initializing [ 179.449514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.469112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.483536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.724214] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.711198] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.136543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.359241] Lustre: lustre-OST0000: new disk, initializing [ 194.363208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 194.367646] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.438756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.227903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.236401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.329102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.095625] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.427869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.583543] Lustre: lustre-OST0001: new disk, initializing [ 212.591656] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.600948] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.685540] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.863038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.880838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.044780] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.163656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.834321] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.292582] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.408374] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.328800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 268.864201] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 271.672255] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 274.286956] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 275.993165] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 16:31:58 EDT 2026 [ 291.679628] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 293.167337] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.289183] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.295288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.306290] Lustre: Skipped 3 previous similar messages [ 312.320042] Lustre: Skipped 3 previous similar messages [ 315.341360] Lustre: server umount lustre-MDT0000 complete [ 319.141115] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813161 with bad export cookie 1227958171141279960 [ 319.143902] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.166272] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.530910] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.533555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.545990] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.550044] LustreError: 6518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 322.604925] Lustre: Skipped 1 previous similar message [ 324.577612] LustreError: 8453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.579314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.601910] LustreError: 8453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 324.618406] Lustre: Skipped 2 previous similar messages [ 325.628553] Lustre: server umount lustre-MDT0001 complete [ 329.914882] Lustre: server umount lustre-OST0000 complete [ 333.487963] Lustre: server umount lustre-OST0001 complete [ 340.401810] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 350.376812] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 384.312408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.556827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.577240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.653765] Lustre: lustre-MDT0000: new disk, initializing [ 384.717399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.735430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 389.021523] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 400.028812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.192166] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 400.287267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 400.293519] Lustre: Skipped 1 previous similar message [ 400.394394] Lustre: lustre-MDT0001: new disk, initializing [ 400.506786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.571645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.589347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 405.239617] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 415.241455] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 421.983679] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 428.833355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.142971] Lustre: lustre-OST0000: new disk, initializing [ 429.153665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.159088] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.256443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.777397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.788890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 430.975154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.171259] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 444.309390] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.364387] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.592612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.298788] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.489308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.687847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.690673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.710039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.721412] Lustre: Skipped 1 previous similar message [ 456.739082] Lustre: Skipped 1 previous similar message [ 461.801483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.816384] Lustre: Skipped 1 previous similar message [ 462.783445] Lustre: server umount lustre-OST0000 complete [ 472.039854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.051758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 472.063262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.070897] Lustre: Skipped 1 previous similar message [ 472.103223] LustreError: Skipped 1 previous similar message [ 473.720361] Lustre: server umount lustre-MDT0000 complete [ 477.160303] LustreError: 17472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.196031] LustreError: 17472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 477.880725] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813320 with bad export cookie 1227958171141281493 [ 477.890903] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 477.895308] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.220630] Lustre: server umount lustre-MDT0001 complete [ 481.640312] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:35:23 (1788813323) [ 487.594645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.972913] LustreError: 19305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.059969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.774789] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 493.541203] LustreError: 19305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.566655] LustreError: 19305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 502.070031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.984442] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 514.440959] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 520.193934] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 526.958715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.343422] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 527.357425] Lustre: Skipped 1 previous similar message [ 535.005896] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 543.921258] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 553.442061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.442061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.442091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.479181] Lustre: Skipped 1 previous similar message [ 558.339659] Lustre: server umount lustre-OST0000 complete [ 563.684195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.692670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.721294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.726291] Lustre: Skipped 3 previous similar messages [ 568.636580] Lustre: server umount lustre-MDT0000 complete [ 568.802417] LustreError: 21137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.821632] LustreError: 21137:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 573.197940] LustreError: 20039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813415 with bad export cookie 1227958171141282375 [ 573.212251] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.217227] LustreError: 20039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 573.670872] Lustre: server umount lustre-MDT0001 complete [ 585.894195] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 588.798243] Key type lgssc unregistered [ 589.081590] LNet: 22836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.091173] LNetError: 22836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.121886] LNet: Removed LNI 192.168.203.145@tcp [ 590.239218] Key type .llcrypt unregistered [ 590.242374] Key type ._llcrypt unregistered [ 615.984066] Key type ._llcrypt registered [ 615.987502] Key type .llcrypt registered [ 616.181701] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 617.814181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.851389] alg: No test for adler32 (adler32-zlib) [ 619.052634] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 619.402332] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 621.136283] Key type lgssc registered [ 622.436694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 630.332256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 630.368150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.805731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.791716] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 642.029275] LustreError: 24246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.046749] LustreError: 24246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 646.801169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 647.141086] LustreError: 24247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.524678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 652.635364] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 661.032209] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 667.686947] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 676.217271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 676.589086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.361163] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 692.619416] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 702.204474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 712.681220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 712.683603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.701243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 712.726755] Lustre: Skipped 1 previous similar message [ 716.660867] Lustre: server umount lustre-OST0000 complete [ 722.914085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.915878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.933958] Lustre: Skipped 1 previous similar message [ 722.947326] Lustre: Skipped 2 previous similar messages [ 727.079693] Lustre: server umount lustre-MDT0000 complete [ 728.033408] LustreError: 24243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 728.055548] LustreError: 24243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 731.110076] LustreError: 24226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813573 with bad export cookie 4880123819192326394 [ 731.126125] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.150230] LustreError: 24226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 731.412329] Lustre: server umount lustre-MDT0001 complete [ 743.183814] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 746.324364] Key type lgssc unregistered [ 746.698190] LNet: 28223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.709277] LNetError: 28223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.724476] LNet: Removed LNI 192.168.203.145@tcp [ 748.163882] Key type .llcrypt unregistered [ 748.167836] Key type ._llcrypt unregistered [ 777.128064] Key type ._llcrypt registered [ 777.132328] Key type .llcrypt registered [ 777.274923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 778.682839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.778327] alg: No test for adler32 (adler32-zlib) [ 779.984542] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 780.283658] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 781.976230] Key type lgssc registered [ 783.234260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 791.568678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 791.593633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.265928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.280194] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 802.801160] LustreError: 29633:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 802.827827] LustreError: 29633:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 807.856805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.910298] LustreError: 29634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.262488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 813.008588] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 821.249641] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 827.637802] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 842.330808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 842.750670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.357685] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 857.279275] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 863.233455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 874.989301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 875.008684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.032101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.564259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.578075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 878.581645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.317532] Lustre: server umount lustre-OST0000 complete [ 885.218945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 885.233952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.252849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.260515] Lustre: Skipped 1 previous similar message [ 888.810168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 888.812540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.476488] Lustre: server umount lustre-MDT0000 complete [ 893.925719] LustreError: 30361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.940683] LustreError: 30361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 894.597386] LustreError: 29615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813737 with bad export cookie 8559701859702412813 [ 894.598860] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.606120] LustreError: 29615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 895.016244] Lustre: server umount lustre-MDT0001 complete [ 905.413076] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 908.205885] Key type lgssc unregistered [ 908.542897] LNet: 33532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.549616] LNetError: 33532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.610597] LNet: Removed LNI 192.168.203.145@tcp [ 910.465376] Key type .llcrypt unregistered [ 910.467898] Key type ._llcrypt unregistered [ 933.436851] Key type ._llcrypt registered [ 933.444943] Key type .llcrypt registered [ 933.593800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 934.755186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 934.924046] alg: No test for adler32 (adler32-zlib) [ 936.026583] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 936.331921] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 938.056272] Key type lgssc registered [ 939.263649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 947.432957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 947.474316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.019295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.490648] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 958.434586] LustreError: 34942:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.456236] LustreError: 34942:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 963.227411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.678609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.271326] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 977.059827] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 984.026657] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 990.077049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.381260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.470342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 995.951520] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1003.314955] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1017.827484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1017.844307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.862822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1019.367694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.372148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1019.382403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.413242] Lustre: Skipped 1 previous similar message [ 1024.484583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1024.490572] Lustre: Skipped 1 previous similar message [ 1029.601805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.614079] Lustre: Skipped 1 previous similar message [ 1031.136149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1031.255138] Lustre: server umount lustre-OST0000 complete [ 1035.239061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1035.253263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.273262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.846088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.217390] Lustre: server umount lustre-MDT0000 complete [ 1044.493791] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788813887 with bad export cookie 14865725152436014131 [ 1044.496405] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.515860] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1044.935501] Lustre: server umount lustre-MDT0001 complete [ 1056.309623] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1059.193283] Key type lgssc unregistered [ 1059.410852] LNet: 38499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.420810] LNetError: 38499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.433575] LNet: Removed LNI 192.168.203.145@tcp [ 1060.393636] Key type .llcrypt unregistered [ 1060.399403] Key type ._llcrypt unregistered [ 1083.919794] Key type ._llcrypt registered [ 1083.930446] Key type .llcrypt registered [ 1084.075284] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1085.176428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.193716] alg: No test for adler32 (adler32-zlib) [ 1086.307499] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1086.639619] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1088.328214] Key type lgssc registered [ 1089.520920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1097.517883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1097.539949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.035926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.384606] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1108.457396] LustreError: 39910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.475340] LustreError: 39910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1111.512188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.832646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.717949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1123.080135] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1128.721270] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.596675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1136.137899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.786629] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1143.381611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1150.633286] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1157.603482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1157.617190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.645584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.631426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.631932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.633685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.677610] Lustre: Skipped 2 previous similar messages [ 1163.514829] Lustre: server umount lustre-OST0000 complete [ 1183.713386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1183.720328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.728824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1189.072849] Lustre: server umount lustre-MDT0000 complete [ 1190.372368] LustreError: 41749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.391450] LustreError: 41749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1192.591820] LustreError: 39889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814035 with bad export cookie 7320650309839491132 [ 1192.595024] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1192.608308] LustreError: 39889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1199.000920] Lustre: server umount lustre-MDT0001 complete [ 1210.299965] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1213.046281] Key type lgssc unregistered [ 1213.315951] LNet: 43613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.331148] LNetError: 43613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.363931] LNet: Removed LNI 192.168.203.145@tcp [ 1214.266528] Key type .llcrypt unregistered [ 1214.273360] Key type ._llcrypt unregistered [ 1235.240565] Key type ._llcrypt registered [ 1235.243115] Key type .llcrypt registered [ 1235.322904] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1236.408172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.432971] alg: No test for adler32 (adler32-zlib) [ 1237.488932] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1237.715141] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1239.359611] Key type lgssc registered [ 1240.476157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1247.819539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1247.837245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.344734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.804601] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1259.496922] LustreError: 45013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1259.513313] LustreError: 45013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1262.155506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.619650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.624254] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1274.676511] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1280.212962] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1286.145848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1286.482979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.880628] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1294.640083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1299.105814] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1305.061303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.063540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.074593] Lustre: Skipped 1 previous similar message [ 1305.084419] Lustre: Skipped 2 previous similar messages [ 1306.074033] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1306.735118] Lustre: server umount lustre-MDT0000 complete [ 1309.753522] LustreError: 44992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814152 with bad export cookie 1178698643875592582 [ 1309.763239] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1309.774740] LustreError: 44992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.180326] LustreError: 45009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.186407] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.209504] Lustre: Skipped 1 previous similar message [ 1310.215292] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1311.198171] LustreError: 45007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.210551] LustreError: 45007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.210580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1314.252232] Lustre: Skipped 1 previous similar message [ 1316.235802] Lustre: server umount lustre-MDT0001 complete [ 1337.304037] Lustre: server umount lustre-OST0000 complete [ 1351.577243] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1354.094319] Key type lgssc unregistered [ 1354.314599] LNet: 48873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.320839] LNetError: 48873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.339778] LNet: Removed LNI 192.168.203.145@tcp [ 1355.031350] Key type .llcrypt unregistered [ 1355.032791] Key type ._llcrypt unregistered [ 1376.666090] Key type ._llcrypt registered [ 1376.668685] Key type .llcrypt registered [ 1376.831880] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1377.650935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.664749] alg: No test for adler32 (adler32-zlib) [ 1378.745698] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1379.027529] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1380.696809] Key type lgssc registered [ 1381.988536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.497544] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.516426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1405.088410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1410.208621] LustreError: 50237:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 1436.899044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.195308] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1449.447258] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1614.265554] Lustre: server umount lustre-OST0000 complete [ 1625.872097] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1627.731545] Key type lgssc unregistered [ 1627.938720] LNet: 51909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.948101] LNetError: 51909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.963629] LNet: Removed LNI 192.168.203.145@tcp [ 1628.549965] Key type .llcrypt unregistered [ 1628.553571] Key type ._llcrypt unregistered [ 1645.764169] Key type ._llcrypt registered [ 1645.767569] Key type .llcrypt registered [ 1645.906050] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1646.700150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.779750] alg: No test for adler32 (adler32-zlib) [ 1647.786856] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1647.992147] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1649.640295] Key type lgssc registered [ 1650.677512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1657.414897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.437695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.808244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.202415] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1668.908868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.154723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1672.413171] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1677.835406] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1681.874593] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1687.456853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.758733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.465344] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1697.915504] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1701.766730] Lustre: server umount lustre-OST0000 complete [ 1704.933569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1704.946191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.957263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.027364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.030337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.596730] Lustre: server umount lustre-MDT0000 complete [ 1713.927983] LustreError: 53286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814556 with bad export cookie 163698899017402528 [ 1713.933450] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.953426] LustreError: 53286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1723.452752] Lustre: server umount lustre-MDT0001 complete [ 1730.797264] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1732.997290] Key type lgssc unregistered [ 1733.210775] LNet: 56833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.215360] LNetError: 56833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.249904] LNet: Removed LNI 192.168.203.145@tcp [ 1734.863448] Key type .llcrypt unregistered [ 1734.865991] Key type ._llcrypt unregistered [ 1750.516555] Key type ._llcrypt registered [ 1750.518774] Key type .llcrypt registered [ 1750.600995] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1751.510730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.527379] alg: No test for adler32 (adler32-zlib) [ 1752.581544] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1752.795770] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1754.425136] Key type lgssc registered [ 1755.304261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1761.685486] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1761.702820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1777.184694] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1782.304712] LustreError: 58204:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 1808.929194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.387317] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1816.989558] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.740757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.063139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1821.131538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1823.408221] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1828.578225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.802825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1831.399737] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1835.932705] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1839.489979] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1841.633279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.638795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.648265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.194592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1844.198879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.202446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.220074] Lustre: Skipped 1 previous similar message [ 1849.318710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.324351] Lustre: Skipped 1 previous similar message [ 1854.435803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1854.445580] Lustre: Skipped 1 previous similar message [ 1854.944591] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1855.031678] Lustre: server umount lustre-OST0000 complete [ 1872.083144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.088413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.092435] Lustre: Skipped 1 previous similar message [ 1878.372299] Lustre: server umount lustre-MDT0000 complete [ 1880.688796] LustreError: 58211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814723 with bad export cookie 14866408887196115082 [ 1880.697319] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.700134] LustreError: 58211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1886.976668] Lustre: server umount lustre-MDT0001 complete [ 1893.393808] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1895.129899] Key type lgssc unregistered [ 1895.304037] LNet: 61944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.310372] LNetError: 61944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.324697] LNet: Removed LNI 192.168.203.145@tcp [ 1895.844192] Key type .llcrypt unregistered [ 1895.846215] Key type ._llcrypt unregistered [ 1910.183319] Key type ._llcrypt registered [ 1910.186582] Key type .llcrypt registered [ 1910.262642] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1910.789771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1910.851484] alg: No test for adler32 (adler32-zlib) [ 1911.826086] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 1911.961279] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1913.576535] Key type lgssc registered [ 1914.213802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.867310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1918.882511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.154094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.699628] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1928.120154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.393333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1931.124392] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1935.859376] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1939.050527] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.472276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.718734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.759694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1947.890264] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1952.592663] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.635754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.636228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1969.638952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1969.644675] Lustre: Skipped 1 previous similar message [ 1972.641778] Lustre: server umount lustre-OST0000 complete [ 1974.753791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1974.755431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.762398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.768859] Lustre: Skipped 1 previous similar message [ 1974.780486] Lustre: Skipped 2 previous similar messages [ 1979.873359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.877922] Lustre: Skipped 1 previous similar message [ 1980.803738] Lustre: server umount lustre-MDT0000 complete [ 1982.883577] LustreError: 63312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814825 with bad export cookie 17748294920443915452 [ 1982.886744] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1982.892322] LustreError: 63312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1983.058650] Lustre: server umount lustre-MDT0001 complete [ 1989.028093] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1990.693956] Key type lgssc unregistered [ 1990.878139] LNet: 66881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.882640] LNetError: 66881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.902696] LNet: Removed LNI 192.168.203.145@tcp [ 1991.414305] Key type .llcrypt unregistered [ 1991.416117] Key type ._llcrypt unregistered [ 2005.831702] Key type ._llcrypt registered [ 2005.834251] Key type .llcrypt registered [ 2005.889713] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2006.365860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2006.433452] alg: No test for adler32 (adler32-zlib) [ 2007.374895] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2007.499338] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2009.105835] Key type lgssc registered [ 2009.764539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2014.656443] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:00:57 (1788814857) [ 2018.719780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2018.739575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.032349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.543979] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2027.634332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.857916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.448666] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2035.041274] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2038.055894] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.850531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.050206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.110716] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2045.162671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2049.096965] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2057.708270] Lustre: Failing over lustre-MDT0000 [ 2057.851756] Lustre: server umount lustre-MDT0000 complete [ 2058.722504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2058.728304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.738951] LustreError: 68574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.750699] LustreError: 68574:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2061.784311] LustreError: 68569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.820106] LustreError: 68575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.827703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.834779] LustreError: 68575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2062.857430] Lustre: Skipped 1 previous similar message [ 2063.335835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.399916] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.494904] Lustre: *** cfs_fail_loc=135, val=0*** [ 2063.558753] LustreError: 71659:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2063.563768] LustreError: 71659:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -2 [ 2063.569764] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2063.569764] [ 2063.582898] LustreError: MGC192.168.203.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2063.592970] LustreError: 71648:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2063.597476] LustreError: 71648:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2063.603036] LustreError: 71648:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2063.606279] LustreError: 71648:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2063.623475] Lustre: server umount lustre-MDT0000 complete [ 2063.626058] LustreError: 71648:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2066.890838] LustreError: 68569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.900862] LustreError: 68569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2067.457921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.529051] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2067.727795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.757202] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2070.278313] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2072.007101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2073.068902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2073.084065] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2073.109501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2078.179770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.181257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.184090] LustreError: Skipped 1 previous similar message [ 2078.193324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.201471] Lustre: Skipped 1 previous similar message [ 2080.942558] Lustre: server umount lustre-OST0000 complete [ 2083.297952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.298663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2083.299975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2083.308478] Lustre: Skipped 2 previous similar messages [ 2088.417923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.421799] Lustre: Skipped 2 previous similar messages [ 2093.538146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.542482] Lustre: Skipped 1 previous similar message [ 2097.120880] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2097.214271] Lustre: server umount lustre-MDT0000 complete [ 2098.658176] LustreError: 68575:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.665087] LustreError: 68575:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2099.078514] LustreError: 68555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814941 with bad export cookie 17691025630019809519 [ 2099.081230] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.083393] LustreError: 68555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2099.233261] Lustre: server umount lustre-MDT0001 complete [ 2104.018383] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2105.337759] Key type lgssc unregistered [ 2105.499968] LNet: 73435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.504028] LNetError: 73435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.518551] LNet: Removed LNI 192.168.203.145@tcp [ 2106.034865] Key type .llcrypt unregistered [ 2106.036957] Key type ._llcrypt unregistered [ 2118.754317] Key type ._llcrypt registered [ 2118.755857] Key type .llcrypt registered [ 2118.818067] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2119.259851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.316332] alg: No test for adler32 (adler32-zlib) [ 2120.217542] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2120.348651] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2121.969154] Key type lgssc registered [ 2122.700122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.964514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.985560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.222412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.385380] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2135.171773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.396460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.430774] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2141.595892] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.838963] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2148.623374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2148.819472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.412724] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2152.940825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2156.724324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2164.372855] Lustre: Failing over lustre-MDT0000 [ 2164.543306] Lustre: server umount lustre-MDT0000 complete [ 2166.252077] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.252936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2166.258051] LustreError: 74821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.258062] LustreError: 74821:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2166.264527] Lustre: Skipped 1 previous similar message [ 2168.295734] LustreError: 74816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.312690] LustreError: 74816:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2170.758370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.863656] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.979807] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2171.065511] LustreError: 77905:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2171.071675] LustreError: 77905:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -9 [ 2171.078727] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2171.078727] [ 2171.086808] LustreError: MGC192.168.203.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2171.096332] LustreError: 77893:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2171.102284] LustreError: 77893:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2171.111920] LustreError: 77893:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2171.117524] LustreError: 77893:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2171.150746] Lustre: server umount lustre-MDT0000 complete [ 2171.160404] LustreError: 77893:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2171.365178] LustreError: 74817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.381348] LustreError: 74817:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2173.377694] LustreError: 75802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.357226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2175.455120] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.619256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.661935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2178.187448] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2178.498598] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2181.104485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2181.133131] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2181.164819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2186.216955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.229103] Lustre: Skipped 1 previous similar message [ 2186.231992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.232811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2186.239464] Lustre: Skipped 1 previous similar message [ 2189.356527] Lustre: server umount lustre-OST0000 complete [ 2196.449084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2196.455216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.460681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.468882] Lustre: Skipped 2 previous similar messages [ 2196.481551] Lustre: Skipped 1 previous similar message [ 2201.569332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.574616] Lustre: Skipped 1 previous similar message [ 2206.176233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2206.286283] Lustre: server umount lustre-MDT0000 complete [ 2206.690936] LustreError: 74816:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.706494] LustreError: 74816:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2208.476505] LustreError: 76662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815051 with bad export cookie 2986569180484461317 [ 2208.480941] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2208.482695] LustreError: 76662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2208.699818] Lustre: server umount lustre-MDT0001 complete [ 2214.776688] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2216.394868] Key type lgssc unregistered [ 2216.602951] LNet: 79681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.609405] LNetError: 79681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.621328] LNet: Removed LNI 192.168.203.145@tcp [ 2217.079589] Key type .llcrypt unregistered [ 2217.081179] Key type ._llcrypt unregistered [ 2223.816381] Key type ._llcrypt registered [ 2223.818303] Key type .llcrypt registered [ 2223.873648] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2224.028977] Key type .llcrypt unregistered [ 2224.030563] Key type ._llcrypt unregistered [ 2238.695754] Key type ._llcrypt registered [ 2238.699500] Key type .llcrypt registered [ 2238.775348] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2239.384026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.398624] alg: No test for adler32 (adler32-zlib) [ 2240.343880] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2240.520116] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2242.168258] Key type lgssc registered [ 2243.047309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.804468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.818923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.095669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.292874] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2256.197407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.488729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.718494] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2263.071195] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2266.337257] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.989271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2270.200360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.221620] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2273.258356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2277.229802] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2288.609693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2288.613112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.619858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.682080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.683929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.687863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2293.488325] Lustre: server umount lustre-OST0000 complete [ 2296.803301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.805929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2296.812253] Lustre: Skipped 1 previous similar message [ 2296.818777] Lustre: Skipped 2 previous similar messages [ 2301.604374] Lustre: server umount lustre-MDT0000 complete [ 2301.920867] LustreError: 81554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.932206] LustreError: 81554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2303.439372] LustreError: 81535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815146 with bad export cookie 6799826968445351846 [ 2303.442841] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.445425] LustreError: 81535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.585114] Lustre: server umount lustre-MDT0001 complete [ 2308.192886] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2309.580175] Key type lgssc unregistered [ 2309.734743] LNet: 85082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.738741] LNetError: 85082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.746888] LNet: Removed LNI 192.168.203.145@tcp [ 2310.140141] Key type .llcrypt unregistered [ 2310.141687] Key type ._llcrypt unregistered [ 2320.438791] Key type ._llcrypt registered [ 2320.440960] Key type .llcrypt registered [ 2320.497865] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2321.048118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2321.069098] alg: No test for adler32 (adler32-zlib) [ 2321.956688] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2322.074935] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2323.688284] Key type lgssc registered [ 2324.162975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.645325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2327.659327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.903252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.717688] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2334.555409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.738359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.531048] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2339.844714] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2342.319122] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2345.283432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.441562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.198790] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2351.722248] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2353.642828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2368.994667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.998839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.007284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.018457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.022449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.031448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.755759] Lustre: server umount lustre-OST0000 complete [ 2375.137587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.139798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2375.145229] Lustre: Skipped 1 previous similar message [ 2375.151416] Lustre: Skipped 2 previous similar messages [ 2379.627380] Lustre: server umount lustre-MDT0000 complete [ 2380.260036] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.273136] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2381.532546] LustreError: 86450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815224 with bad export cookie 16917268477679132148 [ 2381.535744] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.542899] LustreError: 86450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2381.705733] Lustre: server umount lustre-MDT0001 complete [ 2386.829442] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2388.117919] Key type lgssc unregistered [ 2388.273970] LNet: 90018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.282284] LNetError: 90018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.295451] LNet: Removed LNI 192.168.203.145@tcp [ 2388.703489] Key type .llcrypt unregistered [ 2388.705123] Key type ._llcrypt unregistered [ 2399.540694] Key type ._llcrypt registered [ 2399.542159] Key type .llcrypt registered [ 2399.603830] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2399.937787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2399.950669] alg: No test for adler32 (adler32-zlib) [ 2400.822700] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2400.938676] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2402.528482] Key type lgssc registered [ 2402.960575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2406.034186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2406.042042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.226224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.863920] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2412.215972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.404444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2414.270521] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2417.834567] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2420.850771] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2424.099838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.279864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.290694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2430.546948] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.468267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2441.697577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2441.702412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.709180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.722338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.724519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.725020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2446.745172] Lustre: server umount lustre-OST0000 complete [ 2448.352961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2448.355558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.360685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.363145] Lustre: Skipped 1 previous similar message [ 2452.963123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.963465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.373228] Lustre: server umount lustre-MDT0000 complete [ 2455.865234] LustreError: 91384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815298 with bad export cookie 6368176811369192156 [ 2455.867271] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.869852] LustreError: 91384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2455.987308] Lustre: server umount lustre-MDT0001 complete [ 2460.284986] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2461.458407] Key type lgssc unregistered [ 2461.597427] LNet: 94939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.600955] LNetError: 94939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.608446] LNet: Removed LNI 192.168.203.145@tcp [ 2461.967853] Key type .llcrypt unregistered [ 2461.969506] Key type ._llcrypt unregistered [ 2472.441588] Key type ._llcrypt registered [ 2472.443676] Key type .llcrypt registered [ 2472.488960] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2472.916124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2472.976569] alg: No test for adler32 (adler32-zlib) [ 2473.853598] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2473.955322] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2475.544298] Key type lgssc registered [ 2475.990859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2479.342596] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2479.353407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.816403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2499.936707] LustreError: 96298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 2526.560218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.231460] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2532.425230] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2534.840599] Lustre: server umount lustre-OST0000 complete [ 2537.928945] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:09:41 (1788815381) [ 2540.271300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.440443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.904453] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2544.987868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.137728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2546.743505] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2549.771631] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2551.992433] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2554.800122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.956166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.352952] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2560.270906] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2566.250499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2576.866081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2576.871700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.880568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.938780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.939891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2579.944561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.461464] Lustre: server umount lustre-OST0000 complete [ 2585.058417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.059293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.063472] Lustre: Skipped 1 previous similar message [ 2585.066095] Lustre: Skipped 1 previous similar message [ 2590.179178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2590.182229] Lustre: Skipped 2 previous similar messages [ 2595.297218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.300159] Lustre: Skipped 1 previous similar message [ 2597.344124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2597.405528] Lustre: server umount lustre-MDT0000 complete [ 2598.795039] LustreError: 97916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815441 with bad export cookie 12961931042120445495 [ 2598.797587] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.800581] LustreError: 97916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2598.928597] Lustre: server umount lustre-MDT0001 complete [ 2602.968486] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2604.077431] Key type lgssc unregistered [ 2604.221318] LNet: 101645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2604.225068] LNetError: 101645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2604.233382] LNet: Removed LNI 192.168.203.145@tcp [ 2604.578551] Key type .llcrypt unregistered [ 2604.579901] Key type ._llcrypt unregistered [ 2620.967452] Key type ._llcrypt registered [ 2620.969256] Key type .llcrypt registered [ 2621.013880] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2621.367991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.430189] alg: No test for adler32 (adler32-zlib) [ 2622.306678] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2622.415381] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2624.024183] Key type lgssc registered [ 2624.457293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2627.446930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2627.454780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.605629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2630.026869] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2633.152660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.314765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.862523] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2637.667751] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2639.833230] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2642.578306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2642.750113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.864460] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2647.477945] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2653.991609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2664.418897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2664.419749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.422088] LustreError: Skipped 1 previous similar message [ 2664.433982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.436709] Lustre: Skipped 1 previous similar message [ 2667.019855] Lustre: server umount lustre-OST0000 complete [ 2669.025033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.027986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.032157] Lustre: Skipped 1 previous similar message [ 2669.033885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.472804] Lustre: server umount lustre-MDT0000 complete [ 2674.657679] LustreError: 103801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.670675] LustreError: 103801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2676.001628] LustreError: 103783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815518 with bad export cookie 5117293280841180155 [ 2676.006036] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2676.006899] LustreError: 103783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2676.151151] Lustre: server umount lustre-MDT0001 complete [ 2680.420131] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2681.634555] Key type lgssc unregistered [ 2681.773742] LNet: 107351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.777682] LNetError: 107351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2681.787426] LNet: Removed LNI 192.168.203.145@tcp [ 2682.142400] Key type .llcrypt unregistered [ 2682.143858] Key type ._llcrypt unregistered [ 2690.028517] Key type ._llcrypt registered [ 2690.029558] Key type .llcrypt registered [ 2690.069304] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2690.381252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2690.412073] alg: No test for adler32 (adler32-zlib) [ 2691.271774] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2691.365586] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2692.952197] Key type lgssc registered [ 2693.334457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2696.137729] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2696.144662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.584374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2716.704294] LustreError: 108567:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 2743.328172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.419444] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2747.982313] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2750.301294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.392855] LustreError: MGC192.168.203.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2750.401120] LustreError: 109590:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2750.406214] LustreError: 109590:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2750.411622] LustreError: 109590:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2750.444271] Lustre: server umount lustre-MDT0000 complete [ 2750.445806] LustreError: 109590:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2763.780405] Lustre: server umount lustre-OST0000 complete [ 2767.040804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2770.453901] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2783.188360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.272874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2783.285089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2783.321450] Lustre: lustre-MDT0000: new disk, initializing [ 2783.343976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.351605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2784.722797] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2789.039381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.084387] Lustre: 112376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2789.115307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2789.118164] Lustre: Skipped 1 previous similar message [ 2789.154273] Lustre: lustre-MDT0001: new disk, initializing [ 2789.190498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2789.203863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2789.208619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2790.640619] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2794.558958] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2796.478412] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2798.860054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.968399] Lustre: lustre-OST0000: new disk, initializing [ 2798.971710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2798.976246] Lustre: 114343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2799.012104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.027891] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2804.878209] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2806.322440] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2808.826159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2808.831670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2808.864434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2809.456811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2810.889044] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2810.972924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2813.921673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2813.922904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.925479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.928170] Lustre: Skipped 1 previous similar message [ 2813.931500] Lustre: Skipped 1 previous similar message [ 2817.742563] Lustre: server umount lustre-OST0000 complete [ 2819.041731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.042432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.050239] Lustre: Skipped 1 previous similar message [ 2824.161156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.164303] Lustre: Skipped 2 previous similar messages [ 2829.281053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2829.285701] Lustre: Skipped 1 previous similar message [ 2833.376140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2833.378627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2833.386280] Lustre: Skipped 1 previous similar message [ 2833.452612] Lustre: server umount lustre-MDT0000 complete [ 2834.854210] LustreError: 115700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815677 with bad export cookie 7962629708673317796 [ 2834.857619] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.860494] LustreError: 115700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2834.978248] Lustre: server umount lustre-MDT0001 complete [ 2837.880611] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 17:14:40 (1788815680) [ 2839.049254] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2844.356160] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2847.425122] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2859.369178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.454082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2859.464882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2859.502756] Lustre: lustre-MDT0000: new disk, initializing [ 2859.523415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.529165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2860.945517] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2865.439838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.488946] Lustre: 119253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2865.520399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2865.523580] Lustre: Skipped 1 previous similar message [ 2865.563149] Lustre: lustre-MDT0001: new disk, initializing [ 2865.605115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2865.609285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2867.011356] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2870.833651] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2872.786583] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2875.299407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.408976] Lustre: lustre-OST0000: new disk, initializing [ 2875.411167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2875.417381] Lustre: 121221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2875.455575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.459090] Lustre: Skipped 1 previous similar message [ 2876.156691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2876.159924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2876.201775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2877.657993] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2881.679257] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2883.336847] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2883.430248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2885.048672] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2885.132132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2886.627780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.627850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2886.638453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.644680] Lustre: Skipped 1 previous similar message [ 2892.056607] Lustre: server umount lustre-OST0000 complete [ 2896.353390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2896.357406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.364476] Lustre: Skipped 1 previous similar message [ 2899.610421] Lustre: server umount lustre-MDT0000 complete [ 2901.008421] LustreError: 121220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815743 with bad export cookie 7962629708673318664 [ 2901.011535] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.013493] LustreError: 121220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2901.142555] Lustre: server umount lustre-MDT0001 complete [ 2904.460497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.676651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.310490] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2909.768645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.437751] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2914.193672] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2916.279051] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2918.639474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2920.908264] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2923.927424] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2937.329394] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2940.385697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.385835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.389220] LustreError: Skipped 1 previous similar message [ 2940.393650] Lustre: Skipped 1 previous similar message [ 2940.396348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.399919] Lustre: Skipped 5 previous similar messages [ 2944.529646] Lustre: server umount lustre-OST0000 complete [ 2946.016807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.626659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.631820] Lustre: Skipped 1 previous similar message [ 2960.352124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.416891] Lustre: server umount lustre-MDT0000 complete [ 2960.867085] LustreError: 124164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.873705] LustreError: 124164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2961.916337] LustreError: 123417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815804 with bad export cookie 7962629708673319532 [ 2961.919293] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2961.922064] LustreError: 123417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2962.054871] Lustre: server umount lustre-MDT0001 complete [ 2966.136920] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2967.289595] Key type lgssc unregistered [ 2967.424527] LNet: 127084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2967.428119] LNetError: 127084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2967.438157] LNet: Removed LNI 192.168.203.145@tcp [ 2967.797341] Key type .llcrypt unregistered [ 2967.799255] Key type ._llcrypt unregistered [ 2974.747677] Key type ._llcrypt registered [ 2974.749011] Key type .llcrypt registered [ 2974.790680] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2981.348135] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2981.736246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2981.789096] alg: No test for adler32 (adler32-zlib) [ 2982.668528] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 2982.769338] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2984.360245] Key type lgssc registered [ 2984.825704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.738701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.750371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.847449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2998.861475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2998.897963] Lustre: lustre-MDT0000: new disk, initializing [ 2998.925636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.934056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3000.416634] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3004.925297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.977940] Lustre: 130207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3005.003195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3005.006211] Lustre: Skipped 1 previous similar message [ 3005.048682] Lustre: lustre-MDT0001: new disk, initializing [ 3005.093467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3005.109232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3005.114498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3006.599057] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3010.641516] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3012.632224] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3015.057412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.162985] Lustre: lustre-OST0000: new disk, initializing [ 3015.165490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3015.169456] Lustre: 132172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3015.201760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.446423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3015.451574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3015.489845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3017.345398] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3021.327423] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3022.836224] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3022.906262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3024.268768] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3024.352209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3025.893685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.894156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3025.900929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3025.906561] Lustre: Skipped 1 previous similar message [ 3031.012467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.015178] Lustre: Skipped 1 previous similar message [ 3031.130492] Lustre: server umount lustre-OST0000 complete [ 3036.131491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.131539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3036.132422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3036.137768] Lustre: Skipped 2 previous similar messages [ 3036.140459] LustreError: Skipped 1 previous similar message [ 3038.622931] Lustre: server umount lustre-MDT0000 complete [ 3039.913383] LustreError: 130999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815882 with bad export cookie 8510416417021056214 [ 3039.915274] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.919066] LustreError: 130999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3040.036989] Lustre: server umount lustre-MDT0001 complete [ 3042.735989] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:18:05 (1788815885) [ 3044.992673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.171567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.571521] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3049.479581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.611758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.000521] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3053.478205] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3055.338964] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3060.131040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.131966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.134806] Lustre: Skipped 1 previous similar message [ 3060.136667] Lustre: Skipped 1 previous similar message [ 3062.168388] Lustre: server umount lustre-MDT0000 complete [ 3063.474143] LustreError: 134560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815906 with bad export cookie 8510416417021057082 [ 3063.477115] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.478768] LustreError: 134560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3063.587825] Lustre: server umount lustre-MDT0001 complete [ 3066.367581] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:18:29 (1788815909) [ 3068.614738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3084.064513] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3089.184489] LustreError: 136991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 3114.784342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.719377] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3125.958701] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3128.918599] Lustre: server umount lustre-OST0000 complete [ 3137.521078] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:19:39 (1788815979) [ 3144.875234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.281952] LustreError: 138436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.298068] LustreError: 138436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3145.360542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.794313] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3150.817774] LustreError: 138437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.943324] LustreError: 138436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.752336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.204102] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3170.936228] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3174.754128] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3179.231968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3179.466165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3179.469177] Lustre: Skipped 1 previous similar message [ 3183.393456] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3187.769729] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3200.996684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.997647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.012143] Lustre: Skipped 1 previous similar message [ 3201.012484] Lustre: Skipped 1 previous similar message [ 3203.251709] Lustre: server umount lustre-MDT0000 complete [ 3205.149154] LustreError: 138418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816047 with bad export cookie 8510416417021057733 [ 3205.151546] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.156698] LustreError: 138418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3205.345355] Lustre: server umount lustre-MDT0001 complete [ 3217.792990] Lustre: server umount lustre-OST0000 complete [ 3222.776604] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:21:05 (1788816065) [ 3226.479046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.738137] LustreError: 142089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.272539] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3232.230368] LustreError: 142090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.177844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.655867] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3241.146764] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.157373] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3248.106938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.306726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.310544] Lustre: Skipped 2 previous similar messages [ 3251.246855] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3251.373553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3255.126643] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3257.410556] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3257.524984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3259.811788] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3259.921121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3261.923494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3261.931107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.937709] Lustre: Skipped 1 previous similar message [ 3261.941301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3261.943825] Lustre: Skipped 2 previous similar messages [ 3264.994539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.132767] Lustre: server umount lustre-OST0000 complete [ 3270.114494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.126685] Lustre: Skipped 2 previous similar messages [ 3270.132574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3275.438179] Lustre: server umount lustre-MDT0000 complete [ 3277.697716] LustreError: 145196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816120 with bad export cookie 8510416417021058804 [ 3277.698817] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.706402] LustreError: 145196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3277.902221] Lustre: server umount lustre-MDT0001 complete [ 3282.574208] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:22:05 (1788816125) [ 3286.306416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3301.792680] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3306.912454] LustreError: 146243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 3335.781680] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3339.716286] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3343.147834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.399572] LustreError: 146268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.507596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3345.740711] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3350.343796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.569027] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3355.929751] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3358.744271] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3360.951072] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3361.057091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3363.073980] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3363.204646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3365.860996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3365.865264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3365.875895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3365.887280] Lustre: Skipped 8 previous similar messages [ 3370.351293] Lustre: server umount lustre-MDT0000 complete [ 3370.979258] LustreError: 147282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3370.989197] LustreError: 147282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3372.109473] LustreError: 146250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816214 with bad export cookie 8510416417021059826 [ 3372.112682] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3372.118321] LustreError: 146250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3372.263586] Lustre: server umount lustre-MDT0001 complete [ 3384.256514] Lustre: server umount lustre-OST0000 complete [ 3388.108345] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:23:51 (1788816231) [ 3391.365421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.816612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3411.936475] LustreError: 150438:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 3437.537055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.542920] Lustre: Skipped 3 previous similar messages [ 3440.591809] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3444.638360] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3447.980641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3448.185811] LustreError: 150463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.281172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3450.216720] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3453.970746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.912744] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3459.075415] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3461.563959] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3464.886627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.967094] Lustre: lustre-OST0001: new disk, initializing [ 3464.971194] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3464.975484] Lustre: 153311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3467.990954] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3472.906051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3472.913396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3472.991748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3473.026263] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3475.460406] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3475.587698] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3477.850293] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3477.957753] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3479.012161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.016127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.022123] Lustre: Skipped 2 previous similar messages [ 3479.025972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.344623] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3493.392956] Lustre: server umount lustre-OST0000 complete [ 3501.231543] Lustre: server umount lustre-OST0001 complete [ 3509.502937] Lustre: server umount lustre-MDT0000 complete [ 3511.416816] LustreError: 153309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816354 with bad export cookie 8510416417021060778 [ 3511.421109] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3511.422014] LustreError: 153309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3511.569703] Lustre: server umount lustre-MDT0001 complete [ 3523.521141] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:26:06 (1788816366) [ 3524.274259] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3525.028140] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3525.817158] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:26:08 (1788816368) [ 3529.264557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.421749] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3529.436348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3531.477703] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3535.630383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.691323] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3535.711749] Lustre: 156865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3535.746353] LustreError: 156878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.755789] LustreError: 156878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3537.922405] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3541.194332] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3543.779330] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3546.883069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3546.984751] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3548.267181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3549.628135] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3552.847393] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3554.812724] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3554.907100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3556.772548] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3556.870419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3558.368919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3558.374578] LustreError: Skipped 3 previous similar messages [ 3558.379169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3558.384578] Lustre: Skipped 5 previous similar messages [ 3558.387173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3558.390344] Lustre: Skipped 13 previous similar messages [ 3572.192436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3572.268806] Lustre: server umount lustre-OST0000 complete [ 3587.332993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.195879] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3593.716638] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3595.794324] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3597.799234] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3597.806467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3597.806617] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3597.815116] Lustre: Skipped 1 previous similar message [ 3597.935893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3599.704378] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3599.793969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3602.135849] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8dbc48e52000.ost_server_uuid 50 [ 3602.858167] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8dbc48e52000.ost_server_uuid in FULL state after 0 sec [ 3603.002708] LustreError: 156873:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.45@tcp) returned error from blocking AST (req@00000000ea82663c x1875709342721280 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d29d1737/0x761b12a423317004 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.45@tcp remote: 0x725dbf6881e0468d expref: 5 pid: 158870 timeout: 3702 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3603.034720] LustreError: lustre-MDT0000: A client on nid 192.168.203.45@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3603.042124] LustreError: 156863:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.203.45@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d29d1737/0x761b12a423317004 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.45@tcp remote: 0x725dbf6881e0468d expref: 6 pid: 158870 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3611.180570] Lustre: server umount lustre-OST0000 complete [ 3619.370064] Lustre: server umount lustre-MDT0000 complete [ 3621.255917] LustreError: 156860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816464 with bad export cookie 8510416417021061961 [ 3621.258477] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.263462] LustreError: 156860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3627.474909] Lustre: server umount lustre-MDT0001 complete [ 3632.079461] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3633.302993] Key type lgssc unregistered [ 3633.469597] LNet: 162986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3633.475595] LNetError: 162986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3633.489468] LNet: Removed LNI 192.168.203.145@tcp [ 3633.882049] Key type .llcrypt unregistered [ 3633.883834] Key type ._llcrypt unregistered [ 3644.988684] Key type ._llcrypt registered [ 3644.989957] Key type .llcrypt registered [ 3645.041366] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3645.488549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3645.501360] alg: No test for adler32 (adler32-zlib) [ 3646.381617] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 3646.481775] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3648.072234] Key type lgssc registered [ 3648.629355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3652.021894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3652.032557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.233936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3654.947717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3658.449427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.610670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.427724] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3663.814823] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3666.481028] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3669.621647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3669.787829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.488149] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3675.701904] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3677.995154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3688.559793] Lustre: Failing over lustre-MDT0000 [ 3688.732279] Lustre: server umount lustre-MDT0000 complete [ 3689.444330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3689.445188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.445991] LustreError: 164371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.446022] LustreError: 164371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3689.474591] Lustre: Skipped 1 previous similar message [ 3694.563067] LustreError: 165581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.572290] LustreError: 165581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3699.683376] LustreError: 164369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.692110] LustreError: 164369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3699.912651] LustreError: 166973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1788816542 with bad export cookie 4136809051290458178 [ 3703.062121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.131594] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.256215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.285503] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3704.988491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3705.986659] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3705.991656] Lustre: lustre-MDT0000: Denying connection for new client c1c19199-4c8f-462f-8d24-a7304858e6d5 (at 192.168.203.45@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3708.392798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3713.507067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.509509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3713.513742] Lustre: Skipped 2 previous similar messages [ 3718.626593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.630881] Lustre: Skipped 2 previous similar messages [ 3723.745555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3723.751822] Lustre: Skipped 1 previous similar message [ 3727.328119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3727.391420] Lustre: server umount lustre-OST0000 complete [ 3729.009623] LustreError: 168119:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3729.013425] Lustre: 167472:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3729.016783] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3729.024852] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3729.030966] LustreError: 167472:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b7f98a84 x1875710038807936/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3729.041436] LustreError: 167472:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3729.046938] LustreError: 167472:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3729.051913] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 3729.170965] Lustre: server umount lustre-MDT0000 complete [ 3730.850053] LustreError: 164355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816573 with bad export cookie 4136809051290458514 [ 3730.854705] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.855853] LustreError: 164355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3730.856524] LustreError: 163684:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3737.024176] Lustre: server umount lustre-MDT0001 complete [ 3741.550351] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3742.800672] Key type lgssc unregistered [ 3742.951388] LNet: 168869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3742.955262] LNetError: 168869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3742.964585] LNet: Removed LNI 192.168.203.145@tcp [ 3743.339241] Key type .llcrypt unregistered [ 3743.340788] Key type ._llcrypt unregistered [ 3753.966684] Key type ._llcrypt registered [ 3753.968674] Key type .llcrypt registered [ 3754.017775] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3754.376336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3754.409065] alg: No test for adler32 (adler32-zlib) [ 3755.297460] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 3755.411115] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3757.000266] Key type lgssc registered [ 3757.493822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3760.808205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3760.817965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.017383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.855956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3767.508816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.684034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.353533] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3772.819114] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3775.383174] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3778.401025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3778.554881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.881726] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3783.840947] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3786.732369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3791.842243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.843066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3791.849686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3792.866370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3792.872820] Lustre: Skipped 1 previous similar message [ 3793.700433] Lustre: server umount lustre-OST0000 complete [ 3797.471889] Lustre: server umount lustre-MDT0000 complete [ 3797.985331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3797.991052] Lustre: Skipped 1 previous similar message [ 3797.991636] LustreError: 170257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.001885] LustreError: 170257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3799.121187] LustreError: 170989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816641 with bad export cookie 15943610722882645970 [ 3799.128037] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.282264] Lustre: server umount lustre-MDT0001 complete [ 3805.673811] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3806.881949] Key type lgssc unregistered [ 3807.038675] LNet: 173782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3807.042537] LNetError: 173782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3807.051509] LNet: Removed LNI 192.168.203.145@tcp [ 3807.430719] Key type .llcrypt unregistered [ 3807.432781] Key type ._llcrypt unregistered [ 3822.715264] Key type ._llcrypt registered [ 3822.716993] Key type .llcrypt registered [ 3822.769594] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3823.281580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3823.299327] alg: No test for adler32 (adler32-zlib) [ 3824.221115] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 3824.353207] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3825.952421] Key type lgssc registered [ 3826.442308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3829.845756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3829.857650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.044890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.695515] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3836.616568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.800302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3838.574310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3841.967352] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3844.589191] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3847.666532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3847.831960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.582569] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3854.272589] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3856.046465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3871.713451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3871.717318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.727947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3872.738991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3872.739309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3872.743137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.913133] Lustre: server umount lustre-OST0000 complete [ 3877.859314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3877.860707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3877.863247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3877.863253] Lustre: Skipped 1 previous similar message [ 3877.869761] Lustre: Skipped 1 previous similar message [ 3882.665236] Lustre: server umount lustre-MDT0000 complete [ 3882.977334] LustreError: 175473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.985109] LustreError: 175473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3884.255695] LustreError: 175453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816727 with bad export cookie 3274459097391257283 [ 3884.257919] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3884.264210] LustreError: 175453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3884.421282] Lustre: server umount lustre-MDT0001 complete [ 3889.148171] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3890.385847] Key type lgssc unregistered [ 3890.522583] LNet: 179024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3890.527958] LNetError: 179024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3890.539578] LNet: Removed LNI 192.168.203.145@tcp [ 3890.939415] Key type .llcrypt unregistered [ 3890.942533] Key type ._llcrypt unregistered [ 3901.844277] Key type ._llcrypt registered [ 3901.845783] Key type .llcrypt registered [ 3901.892023] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3902.454061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3902.466466] alg: No test for adler32 (adler32-zlib) [ 3903.360832] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 3903.473770] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3905.072242] Key type lgssc registered [ 3905.553810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3910.744114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3910.757028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.950708] Lustre: *** cfs_fail_loc=135, val=0*** [ 3912.002784] LustreError: 180520:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3912.006918] LustreError: 180520:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -2 [ 3912.012102] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3912.012102] [ 3912.023506] LustreError: MGC192.168.203.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3912.029376] LustreError: 180481:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3912.034157] LustreError: 180481:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3912.037209] LustreError: 180481:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3912.041472] LustreError: 180481:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3912.089714] Lustre: server umount lustre-MDT0000 complete [ 3912.092337] LustreError: 180481:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3915.267836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.484968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3917.401195] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3921.498276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.686085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3923.690779] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3927.306374] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3930.155955] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3932.129514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3932.134948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.144459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.154673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.154995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.165453] Lustre: Skipped 1 previous similar message [ 3937.381932] Lustre: server umount lustre-MDT0000 complete [ 3938.274652] LustreError: 180890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.284331] LustreError: 180890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3939.141673] LustreError: 180874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816781 with bad export cookie 452333256911015058 [ 3939.144091] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3939.148184] LustreError: 180874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3939.296702] Lustre: server umount lustre-MDT0001 complete [ 3944.074788] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3945.348080] Key type lgssc unregistered [ 3945.500573] LNet: 183214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3945.504941] LNetError: 183214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3945.512554] LNet: Removed LNI 192.168.203.145@tcp [ 3945.915635] Key type .llcrypt unregistered [ 3945.917570] Key type ._llcrypt unregistered [ 3956.257110] Key type ._llcrypt registered [ 3956.260145] Key type .llcrypt registered [ 3956.318233] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3956.480617] Key type .llcrypt unregistered [ 3956.482921] Key type ._llcrypt unregistered [ 3964.988637] Key type ._llcrypt registered [ 3964.990833] Key type .llcrypt registered [ 3965.034643] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3965.480505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.521499] alg: No test for adler32 (adler32-zlib) [ 3966.417931] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 3966.533851] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3968.136192] Key type lgssc registered [ 3968.612984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3971.903155] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3971.913279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.360498] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3992.480425] LustreError: 185313:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 4019.104188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.946554] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4025.395750] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4028.731517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.970396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.001370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4030.831305] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4034.018847] LustreError: 185339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.025394] LustreError: 185339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4034.722825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.919193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4036.710873] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4040.144350] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4042.654649] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4043.859803] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4049.376968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.381073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4049.388601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.401913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.410837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.419209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.427730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.432764] Lustre: Skipped 1 previous similar message [ 4052.651092] Lustre: server umount lustre-OST0000 complete [ 4055.521462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.525340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.534743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.458606] Lustre: server umount lustre-MDT0000 complete [ 4061.666038] LustreError: 186353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4061.674296] LustreError: 186353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4062.129875] LustreError: 187086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816904 with bad export cookie 4698462586372176048 [ 4062.131917] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.140758] LustreError: 187086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4062.289855] Lustre: server umount lustre-MDT0001 complete [ 4066.813651] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4068.087719] Key type lgssc unregistered [ 4068.241418] LNet: 189128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4068.245549] LNetError: 189128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4068.256285] LNet: Removed LNI 192.168.203.145@tcp [ 4068.650298] Key type .llcrypt unregistered [ 4068.652851] Key type ._llcrypt unregistered [ 4080.062220] Key type ._llcrypt registered [ 4080.063915] Key type .llcrypt registered [ 4080.108508] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4080.564264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4080.572289] alg: No test for adler32 (adler32-zlib) [ 4081.458046] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4081.571056] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4083.168600] Key type lgssc registered [ 4083.674848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4087.373892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4087.389813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4088.616486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4090.272858] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4093.954808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.126372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4095.865193] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4099.148525] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4101.660499] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4104.831767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4105.019941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.670422] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4110.789684] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4114.218350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4118.800463] Lustre: Failing over lustre-MDT0000 [ 4118.933502] Lustre: server umount lustre-MDT0000 complete [ 4120.033324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4120.037493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.043897] LustreError: 190514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.051127] LustreError: 190514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4122.574989] LustreError: 190509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.618596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.618972] LustreError: 190513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.633348] LustreError: 190513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4127.684775] LustreError: 190509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.802470] LustreError: 190508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.810037] LustreError: 190508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4132.953303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.023579] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.159806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.186443] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4134.857938] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4136.877390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4137.922101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4138.471381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4138.482891] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4138.506852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4140.396312] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4148.709527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4148.711040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4148.711050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4148.715664] Lustre: Skipped 2 previous similar messages [ 4152.031866] Lustre: server umount lustre-OST0000 complete [ 4153.826689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.827608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4153.828507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.828511] Lustre: Skipped 1 previous similar message [ 4153.837212] Lustre: Skipped 1 previous similar message [ 4158.945264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4158.949049] Lustre: Skipped 2 previous similar messages [ 4159.713381] Lustre: server umount lustre-MDT0000 complete [ 4161.176237] LustreError: 190494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817003 with bad export cookie 1954547906367929725 [ 4161.179157] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4161.183899] LustreError: 190494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4161.325377] Lustre: server umount lustre-MDT0001 complete [ 4165.861925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4167.043631] Key type lgssc unregistered [ 4167.177035] LNet: 195342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4167.181341] LNetError: 195342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4167.189627] LNet: Removed LNI 192.168.203.145@tcp [ 4167.555122] Key type .llcrypt unregistered [ 4167.556426] Key type ._llcrypt unregistered [ 4178.362461] Key type ._llcrypt registered [ 4178.363940] Key type .llcrypt registered [ 4178.408850] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4178.827703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4178.837704] alg: No test for adler32 (adler32-zlib) [ 4179.711640] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4179.816203] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4181.408161] Key type lgssc registered [ 4181.832801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4184.886406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4184.896948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.066462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.614249] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4190.932249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.096403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4192.732389] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4195.680900] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4197.906635] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4200.378389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4200.504457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.656267] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4205.477475] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4210.726035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4212.648194] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4215.154952] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4225.477986] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4231.138310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.140278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.144334] LustreError: Skipped 1 previous similar message [ 4231.150298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.154058] Lustre: Skipped 1 previous similar message [ 4236.259308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4236.262977] Lustre: Skipped 1 previous similar message [ 4236.770506] Lustre: server umount lustre-OST0000 complete [ 4241.377741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.378464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.382253] Lustre: Skipped 2 previous similar messages [ 4244.310578] Lustre: server umount lustre-MDT0000 complete [ 4245.685284] LustreError: 196708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817088 with bad export cookie 12970009890932139864 [ 4245.687782] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4245.691157] LustreError: 196708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4245.821878] Lustre: server umount lustre-MDT0001 complete [ 4250.170491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4251.308268] Key type lgssc unregistered [ 4251.442463] LNet: 200570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4251.445250] LNetError: 200570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4251.452586] LNet: Removed LNI 192.168.203.145@tcp [ 4251.785056] Key type .llcrypt unregistered [ 4251.786656] Key type ._llcrypt unregistered [ 4260.287607] Key type ._llcrypt registered [ 4260.289163] Key type .llcrypt registered [ 4260.334696] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4260.927426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4260.942922] alg: No test for adler32 (adler32-zlib) [ 4261.832387] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4261.933632] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4263.520173] Key type lgssc registered [ 4263.962055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4267.092699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4267.103186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.298249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.926869] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4273.171940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.307186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4274.872551] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4277.571637] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.698558] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4282.186987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.332445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4284.488322] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4287.214508] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4293.610739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4299.259405] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4308.572481] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4315.635931] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4326.102658] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4334.564623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.566273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4334.567235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.570624] Lustre: Skipped 1 previous similar message [ 4334.573213] LustreError: Skipped 1 previous similar message [ 4339.681928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.689548] Lustre: Skipped 2 previous similar messages [ 4344.804724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4344.807291] Lustre: Skipped 1 previous similar message [ 4347.360424] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4347.424315] Lustre: server umount lustre-OST0000 complete [ 4349.921510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.922275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.925313] Lustre: Skipped 1 previous similar message [ 4355.041200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4355.045023] Lustre: Skipped 2 previous similar messages [ 4363.232088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4363.291782] Lustre: server umount lustre-MDT0000 complete [ 4364.564273] LustreError: 204511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817207 with bad export cookie 6347759726203628109 [ 4364.567393] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4364.568920] LustreError: 204511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4364.682962] Lustre: server umount lustre-MDT0001 complete [ 4368.609241] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4369.678223] Key type lgssc unregistered [ 4369.811554] LNet: 207282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4369.814270] LNetError: 207282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4369.827428] LNet: Removed LNI 192.168.203.145@tcp [ 4370.146830] Key type .llcrypt unregistered [ 4370.147961] Key type ._llcrypt unregistered [ 4379.698508] Key type ._llcrypt registered [ 4379.700109] Key type .llcrypt registered [ 4379.741302] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4380.069219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4380.081825] alg: No test for adler32 (adler32-zlib) [ 4380.947235] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4381.037445] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4382.616194] Key type lgssc registered [ 4382.994475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4385.676400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4385.683038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.838070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.337901] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4391.383326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.511223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4392.943861] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4395.509081] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.475819] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4399.803285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4399.960673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.960436] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4404.551389] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4412.262821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4422.625776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4422.626059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.627996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4422.629813] LustreError: Skipped 1 previous similar message [ 4422.634217] Lustre: Skipped 1 previous similar message [ 4424.476459] Lustre: server umount lustre-OST0000 complete [ 4427.232782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4427.235031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.238807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.240492] Lustre: Skipped 1 previous similar message [ 4431.956398] Lustre: server umount lustre-MDT0000 complete [ 4432.865870] LustreError: 208668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.871206] LustreError: 208668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4433.214526] LustreError: 211329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817275 with bad export cookie 6468382864687458122 [ 4433.219283] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4433.219646] LustreError: 211329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4433.338899] Lustre: server umount lustre-MDT0001 complete [ 4436.997828] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4438.030810] Key type lgssc unregistered [ 4438.163567] LNet: 212366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4438.167378] LNetError: 212366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4438.177311] LNet: Removed LNI 192.168.203.145@tcp [ 4438.539821] Key type .llcrypt unregistered [ 4438.541230] Key type ._llcrypt unregistered [ 4446.505978] Key type ._llcrypt registered [ 4446.507564] Key type .llcrypt registered [ 4446.556320] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4446.867570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4446.875981] alg: No test for adler32 (adler32-zlib) [ 4447.742736] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4447.847044] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4449.432176] Key type lgssc registered [ 4449.761442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4452.258462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4452.264372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4453.400030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.741549] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4457.671661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.801958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4459.108852] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4461.794080] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4463.717859] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4466.083140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4466.210886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.232100] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4470.554660] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4478.501284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4488.678646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.679229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4488.686437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.690910] LustreError: Skipped 1 previous similar message [ 4488.696043] Lustre: Skipped 1 previous similar message [ 4492.396845] Lustre: server umount lustre-OST0000 complete [ 4498.916461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4498.917394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.920427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.945044] Lustre: Skipped 2 previous similar messages [ 4504.033431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4504.037504] Lustre: Skipped 2 previous similar messages [ 4509.153841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4509.157780] Lustre: Skipped 1 previous similar message [ 4510.176253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4510.443572] Lustre: server umount lustre-MDT0000 complete [ 4514.278824] LustreError: 215036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.288574] LustreError: 215036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4515.400959] LustreError: 214337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817358 with bad export cookie 15937510490413273317 [ 4515.415506] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4515.429269] LustreError: 214337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4516.057533] Lustre: server umount lustre-MDT0001 complete [ 4543.382238] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4548.875867] Key type lgssc unregistered [ 4549.185850] LNet: 217332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4549.192737] LNetError: 217332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4549.204513] LNet: Removed LNI 192.168.203.145@tcp [ 4550.092931] Key type .llcrypt unregistered [ 4550.096714] Key type ._llcrypt unregistered [ 4577.420191] Key type ._llcrypt registered [ 4577.434284] Key type .llcrypt registered [ 4577.617742] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4579.627068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4580.048949] alg: No test for adler32 (adler32-zlib) [ 4581.337126] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4581.800697] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4583.616226] Key type lgssc registered [ 4585.164370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4594.454830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4594.519148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.438663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.351903] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4606.946507] LustreError: 218595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.990433] LustreError: 218595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4607.973969] LustreError: 218596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.128979] LustreError: 218595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.068606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.677491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4622.981740] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4630.298942] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4635.846328] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4641.423377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4641.665668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.685507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4646.422688] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4652.731278] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4661.456026] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4661.507637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4661.690287] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4668.183100] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4675.516177] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4677.026622] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4688.714364] Lustre: Permanently deactivating lustre-OST0001 [ 4688.741748] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4694.648990] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4701.580759] Lustre: Permanently reactivating lustre-OST0001 [ 4701.604306] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4701.615952] Lustre: Skipped 2 previous similar messages [ 4703.699831] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.716732] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4703.723179] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4703.750858] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4703.776384] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4713.954654] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4713.955663] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.968776] LustreError: Skipped 1 previous similar message [ 4713.970712] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4713.981773] Lustre: Skipped 2 previous similar messages [ 4713.998435] Lustre: Skipped 1 previous similar message [ 4714.454962] Lustre: server umount lustre-OST0001 complete [ 4714.985556] LustreError: 220525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.015143] LustreError: 220525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4719.585471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4719.591931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.594461] LustreError: Skipped 1 previous similar message [ 4719.595254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.622604] Lustre: Skipped 1 previous similar message [ 4720.104204] LustreError: 220422:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.136965] LustreError: 220422:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4724.844463] Lustre: server umount lustre-OST0000 complete [ 4730.343634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.351256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4730.352210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4730.365749] Lustre: Skipped 1 previous similar message [ 4730.405729] Lustre: Skipped 4 previous similar messages [ 4735.050276] Lustre: server umount lustre-MDT0000 complete [ 4735.464504] LustreError: 219829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.483969] LustreError: 219829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4738.590944] LustreError: 218575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817581 with bad export cookie 15422264114090797816 [ 4738.597686] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4738.601912] LustreError: 218575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4739.032677] Lustre: server umount lustre-MDT0001 complete [ 4749.857804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4752.909803] Key type lgssc unregistered [ 4753.189623] LNet: 223643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4753.198933] LNetError: 223643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4753.219740] LNet: Removed LNI 192.168.203.145@tcp [ 4754.265882] Key type .llcrypt unregistered [ 4754.268482] Key type ._llcrypt unregistered [ 4794.090978] Key type ._llcrypt registered [ 4794.098487] Key type .llcrypt registered [ 4794.258963] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4795.938257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4795.978374] alg: No test for adler32 (adler32-zlib) [ 4797.083176] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 4797.343910] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4799.069038] Key type lgssc registered [ 4800.250725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4807.998511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4808.030696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.413231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4809.449857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4809.631200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.197873] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4824.486346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4824.673962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4824.723598] Lustre: 225844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4824.868024] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4825.173105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4832.336362] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4842.499335] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4849.252965] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4857.784342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4858.001197] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4858.223128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.231648] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4867.867617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 4876.249837] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4880.346889] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4886.142268] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4894.933896] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4906.648086] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4915.403419] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4932.026044] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4932.040315] Lustre: Skipped 1 previous similar message [ 4964.858050] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4964.862797] Lustre: Skipped 3 previous similar messages [ 5031.167964] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5031.171989] Lustre: Skipped 7 previous similar messages [ 5067.745466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5067.747942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.769214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5067.780454] Lustre: Skipped 1 previous similar message [ 5069.972602] Lustre: server umount lustre-OST0000 complete [ 5075.938701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5075.950257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5075.970736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5075.981601] Lustre: Skipped 1 previous similar message [ 5077.990562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.996633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.021183] Lustre: Skipped 1 previous similar message [ 5080.400126] Lustre: server umount lustre-MDT0000 complete [ 5085.100635] LustreError: 225837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817927 with bad export cookie 2593338868997760029 [ 5085.103665] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.114537] LustreError: 225837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5085.388997] Lustre: server umount lustre-MDT0001 complete [ 5094.945345] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5098.768448] Key type lgssc unregistered [ 5098.991427] LNet: 230676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5098.999021] LNetError: 230676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5099.015101] LNet: Removed LNI 192.168.203.145@tcp [ 5099.795048] Key type .llcrypt unregistered [ 5099.796508] Key type ._llcrypt unregistered [ 5130.411489] Key type ._llcrypt registered [ 5130.419860] Key type .llcrypt registered [ 5130.593872] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5132.159965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5132.240235] alg: No test for adler32 (adler32-zlib) [ 5134.121487] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 5134.427954] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5136.144808] Key type lgssc registered [ 5138.273774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5149.076436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5149.095852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.631454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.179543] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5160.929923] LustreError: 232089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.960601] LustreError: 232089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5165.883669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.050596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5166.259230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5170.628547] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5177.012495] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5182.228607] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5187.831143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5188.144555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.704839] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5195.313638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5199.719382] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5214.498573] Lustre: server umount lustre-OST0000 complete [ 5217.761642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5217.766645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.782051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.836981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.838316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.865239] Lustre: Skipped 1 previous similar message [ 5225.954878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5225.962836] Lustre: Skipped 1 previous similar message [ 5231.073375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.080022] Lustre: Skipped 1 previous similar message [ 5232.096265] Lustre: 231399:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788818058/real 1788818058] req@000000001ea7ef7e x1875711599083520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788818074 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5232.097075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5232.141134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.302228] Lustre: server umount lustre-MDT0000 complete [ 5233.122164] Lustre: 231401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788818059/real 1788818059] req@00000000551d5edd x1875711599083776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788818075 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5235.156580] LustreError: 233931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818077 with bad export cookie 1599965933476175430 [ 5235.166530] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5235.177049] LustreError: 233931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5235.441230] Lustre: server umount lustre-MDT0001 complete [ 5244.988634] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5248.278932] Key type lgssc unregistered [ 5248.672020] LNet: 235773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5248.687824] LNetError: 235773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5248.712265] LNet: Removed LNI 192.168.203.145@tcp [ 5249.965690] Key type .llcrypt unregistered [ 5249.972703] Key type ._llcrypt unregistered [ 5276.120875] Key type ._llcrypt registered [ 5276.123109] Key type .llcrypt registered [ 5276.219760] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5276.982788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5277.127834] alg: No test for adler32 (adler32-zlib) [ 5278.348326] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 5278.611649] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5280.288968] Key type lgssc registered [ 5281.576727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5551.417931] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5555.060099] Key type lgssc unregistered [ 5555.417952] LNet: 237793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5555.424881] LNetError: 237793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5555.436733] LNet: Removed LNI 192.168.203.145@tcp [ 5556.349288] Key type .llcrypt unregistered [ 5556.358948] Key type ._llcrypt unregistered [ 5587.660503] Key type ._llcrypt registered [ 5587.662386] Key type .llcrypt registered [ 5587.838394] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5604.304260] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5605.212150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5605.245560] alg: No test for adler32 (adler32-zlib) [ 5606.341589] Lustre: Lustre: Build Version: 2.17.58_41_g50dbf09 [ 5606.566510] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5608.264353] Key type lgssc registered [ 5609.366592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5636.161935] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5359 sec ========== 18:01:18 (1788818478) [ 5638.114666] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:01:20 (1788818480) === [ 5641.013413] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:01:23 (1788818483) === [ 5675.447576] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5678.230955] Key type lgssc unregistered [ 5678.585872] LNet: 243036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5678.591357] LNetError: 243036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5678.608887] LNet: Removed LNI 192.168.203.145@tcp [ 5679.541918] Key type .llcrypt unregistered [ 5679.545412] Key type ._llcrypt unregistered