[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479242125 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002372] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007692] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012035] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.015026] *** VALIDATE selinux *** [ 0.025365] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029696] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031042] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033134] *** VALIDATE tmpfs *** [ 0.035118] *** VALIDATE proc *** [ 0.036244] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.039204] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042031] Spectre V2 : User space: Vulnerable [ 0.043015] Speculative Store Bypass: Vulnerable [ 0.046324] debug: unmapping init [mem 0xffffffff8b459000-0xffffffff8b460fff] [ 0.048988] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050025] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.052013] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055013] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057337] rcu: Hierarchical SRCU implementation. [ 0.059411] smp: Bringing up secondary CPUs ... [ 0.060562] x86: Booting SMP configuration: [ 0.061023] .... node #0, CPUs: #1 #2 #3 [ 0.065092] smp: Brought up 1 node, 4 CPUs [ 0.067017] smpboot: Max logical packages: 1 [ 0.068013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.184446] node 0 deferred pages initialised in 114ms [ 0.187104] devtmpfs: initialized [ 0.188249] x86/mm: Memory block size: 128MB [ 0.190923] gcov: version magic: 0x41383552 [ 0.192232] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.193103] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.194268] pinctrl core: initialized pinctrl subsystem [ 0.195189] [ 0.195557] ************************************************************* [ 0.196014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197015] ** ** [ 0.198014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199015] ** ** [ 0.200015] ** This means that this kernel is built to expose internal ** [ 0.201015] ** IOMMU data structures, which may compromise security on ** [ 0.202023] ** your system. ** [ 0.203017] ** ** [ 0.204025] ** If you see this message and you are not debugging the ** [ 0.205012] ** kernel, report this immediately to your vendor! ** [ 0.206019] ** ** [ 0.207014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208015] ************************************************************* [ 0.209825] NET: Registered protocol family 16 [ 0.210478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213451] cpuidle: using governor menu [ 0.215707] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218440] PCI: Using configuration type 1 for base access [ 0.219134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235198] cryptd: max_cpu_qlen set to 1000 [ 0.238018] ACPI: Added _OSI(Module Device) [ 0.239013] ACPI: Added _OSI(Processor Device) [ 0.240013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246638] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254417] ACPI: Interpreter enabled [ 0.255056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256013] ACPI: Using IOAPIC for interrupt routing [ 0.257099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268502] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273436] acpiphp: Slot [2] registered [ 0.274115] acpiphp: Slot [5] registered [ 0.275109] acpiphp: Slot [6] registered [ 0.276116] acpiphp: Slot [7] registered [ 0.277118] acpiphp: Slot [8] registered [ 0.278149] acpiphp: Slot [9] registered [ 0.279125] acpiphp: Slot [10] registered [ 0.280112] acpiphp: Slot [3] registered [ 0.281097] acpiphp: Slot [4] registered [ 0.282128] acpiphp: Slot [11] registered [ 0.283000] acpiphp: Slot [12] registered [ 0.284127] acpiphp: Slot [13] registered [ 0.286195] acpiphp: Slot [14] registered [ 0.287107] acpiphp: Slot [15] registered [ 0.288482] acpiphp: Slot [16] registered [ 0.290108] acpiphp: Slot [17] registered [ 0.291129] acpiphp: Slot [18] registered [ 0.292114] acpiphp: Slot [19] registered [ 0.293161] acpiphp: Slot [20] registered [ 0.295197] acpiphp: Slot [21] registered [ 0.297102] acpiphp: Slot [22] registered [ 0.298104] acpiphp: Slot [23] registered [ 0.299131] acpiphp: Slot [24] registered [ 0.300112] acpiphp: Slot [25] registered [ 0.302111] acpiphp: Slot [26] registered [ 0.303122] acpiphp: Slot [27] registered [ 0.304101] acpiphp: Slot [28] registered [ 0.306026] acpiphp: Slot [29] registered [ 0.307140] acpiphp: Slot [30] registered [ 0.309128] acpiphp: Slot [31] registered [ 0.310077] PCI host bridge to bus 0000:00 [ 0.311021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328175] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332408] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346017] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355502] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360203] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363164] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367146] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.373018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402747] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452607] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460050] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.467019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.488018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507292] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.525041] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.568634] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.577014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.598019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.607504] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617062] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.638017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.649473] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.657017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.696921] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.699401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.704355] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.712132] iommu: Default domain type: Passthrough [ 0.714596] SCSI subsystem initialized [ 0.716165] ACPI: bus type USB registered [ 0.718128] usbcore: registered new interface driver usbfs [ 0.720116] usbcore: registered new interface driver hub [ 0.722111] usbcore: registered new device driver usb [ 0.724228] pps_core: LinuxPPS API ver. 1 registered [ 0.726015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729064] PTP clock support registered [ 0.731160] EDAC MC: Ver: 3.0.0 [ 0.733147] PCI: Using ACPI for IRQ routing [ 0.734868] NetLabel: Initializing [ 0.736012] NetLabel: domain hash size = 128 [ 0.738010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.739089] NetLabel: unlabeled traffic allowed by default [ 0.741133] vgaarb: loaded [ 0.743257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.749348] clocksource: Switched to clocksource kvm-clock [ 0.863528] VFS: Disk quotas dquot_6.6.0 [ 0.865146] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867838] *** VALIDATE ramfs *** [ 0.869063] *** VALIDATE hugetlbfs *** [ 0.870792] pnp: PnP ACPI init [ 0.873679] pnp: PnP ACPI: found 6 devices [ 0.892677] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896322] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.898720] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901160] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903900] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906417] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909819] NET: Registered protocol family 2 [ 0.912810] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917749] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921243] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.929533] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.934860] TCP: Hash tables configured (established 65536 bind 65536) [ 0.938953] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.943607] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948369] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951578] NET: Registered protocol family 1 [ 0.955230] RPC: Registered named UNIX socket transport module. [ 0.956916] RPC: Registered udp transport module. [ 0.958314] RPC: Registered tcp transport module. [ 0.959612] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.961453] NET: Registered protocol family 44 [ 0.962595] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964083] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966559] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968524] PCI: CLS 0 bytes, default 64 [ 0.970827] Unpacking initramfs... [ 2.435151] debug: unmapping init [mem 0xffff9a80fcc54000-0xffff9a80fffbffff] [ 2.438844] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.440682] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.443491] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.960095] Initialise system trusted keyrings [ 2.961804] Key type blacklist registered [ 2.963775] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.972574] zbud: loaded [ 2.975685] *** VALIDATE nfs *** [ 2.976675] *** VALIDATE nfs4 *** [ 2.978822] pstore: using deflate compression [ 2.982787] Platform Keyring initialized [ 3.088898] NET: Registered protocol family 38 [ 3.090635] Key type asymmetric registered [ 3.092178] Asymmetric key parser 'x509' registered [ 3.094315] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.097427] io scheduler mq-deadline registered [ 3.099370] io scheduler kyber registered [ 3.101172] io scheduler bfq registered [ 3.103142] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.106339] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.109703] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.113257] ACPI: Power Button [PWRF] [ 3.118865] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.125184] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.140514] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.150170] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.167740] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.199349] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.228955] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.234965] Non-volatile memory driver v1.3 [ 3.236615] Linux agpgart interface v0.103 [ 3.273552] virtio_blk virtio1: [vda] 145960 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.275955] vda: detected capacity change from 0 to 74731520 [ 3.290109] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.292614] vdb: detected capacity change from 0 to 1073741824 [ 3.305658] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.309579] vdc: detected capacity change from 0 to 2621440000 [ 3.326292] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.330380] vdd: detected capacity change from 0 to 2621440000 [ 3.347707] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.352332] vde: detected capacity change from 0 to 4294967296 [ 3.368246] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.370940] vdf: detected capacity change from 0 to 4294967296 [ 3.376481] libphy: Fixed MDIO Bus: probed [ 3.383761] usbcore: registered new interface driver usbserial_generic [ 3.386846] usbserial: USB Serial support registered for generic [ 3.389539] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.393830] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.395719] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.398468] mousedev: PS/2 mouse device common for all mice [ 3.401916] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.406719] rtc_cmos 00:05: RTC can wake from S4 [ 3.410577] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.413354] rtc_cmos 00:05: registered as rtc0 [ 3.417454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.417843] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.425286] intel_pstate: CPU model not supported [ 3.428494] hid: raw HID events driver (C) Jiri Kosina [ 3.430647] usbcore: registered new interface driver usbhid [ 3.433112] usbhid: USB HID core driver [ 3.434650] drop_monitor: Initializing network drop monitor service [ 3.437713] Initializing XFRM netlink socket [ 3.440421] NET: Registered protocol family 10 [ 3.443514] Segment Routing with IPv6 [ 3.444891] NET: Registered protocol family 17 [ 3.447242] mpls_gso: MPLS GSO support [ 3.453693] RAS: Correctable Errors collector initialized. [ 3.455248] AVX version of gcm_enc/dec engaged. [ 3.456714] AES CTR mode by8 optimization enabled [ 3.542803] sched_clock: Marking stable (3542780408, 0)->(4503929184, -961148776) [ 3.546405] registered taskstats version 1 [ 3.548428] Loading compiled-in X.509 certificates [ 3.550725] zswap: loaded using pool lzo/zbud [ 3.576565] Key type big_key registered [ 3.592357] Key type encrypted registered [ 3.595628] ima: No TPM chip found, activating TPM-bypass! [ 3.597583] ima: Allocated hash algorithm: sha1 [ 3.599824] ima: No architecture policies found [ 3.602171] evm: Initialising EVM extended attributes: [ 3.605508] evm: security.selinux [ 3.607108] evm: security.ima [ 3.608644] evm: security.capability [ 3.610448] evm: HMAC attrs: 0x1 [ 3.613196] rtc_cmos 00:05: setting system clock to 2026-09-08 21:35:22 UTC (1788903322) [ 3.620880] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 3.625049] debug: unmapping init [mem 0xffffffff8b182000-0xffffffff8b458fff] [ 3.637282] Write protecting the kernel read-only data: 28672k [ 3.641571] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.645437] debug: unmapping init [mem 0xffffffff8a114000-0xffffffff8a1fffff] [ 3.688326] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.696725] systemd[1]: Detected virtualization kvm. [ 3.698651] systemd[1]: Detected architecture x86-64. [ 3.700627] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728570] systemd[1]: No hostname configured. [ 3.730996] systemd[1]: Set hostname to . [ 3.733821] random: systemd: uninitialized urandom read (16 bytes read) [ 3.736034] systemd[1]: Initializing machine ID from random generator. [ 3.922162] random: systemd: uninitialized urandom read (16 bytes read) [ 3.924897] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.929701] random: systemd: uninitialized urandom read (16 bytes read) [ 3.932168] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.936384] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.639734] device-mapper: uevent: version 1.0.3 [ 4.642361] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.385387] virtio_net virtio0 ens2: renamed from eth0 [ 5.394243] random: fast init done [ 5.471211] scsi host0: ata_piix [ 5.577582] scsi host1: ata_piix [ 5.579226] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.582028] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.302473] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.116473] random: crng init done [ 10.117828] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.673817] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.879231] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.151308] SELinux: Disabled at runtime. [ 12.213380] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.221869] systemd[1]: Detected virtualization kvm. [ 12.223983] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.727489] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.730720] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.735706] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.739858] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.742995] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.750718] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.754278] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ 12.865781] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.232658] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.562061] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.586243] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.745480] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.758751] EDAC sbridge: Ver: 1.1.2 [ 15.547406] Key type dns_resolver registered [ 15.872460] NFS: Registering the id_resolver key type [ 15.874508] Key type id_resolver registered [ 15.876539] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg419-server login: [ 47.022014] hrtimer: interrupt took 4330976 ns [ 56.812695] libcfs: loading out-of-tree module taints kernel. [ 56.884413] Key type ._llcrypt registered [ 56.885985] Key type .llcrypt registered [ 57.020950] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 80.728670] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 82.232234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.248948] alg: No test for adler32 (adler32-zlib) [ 83.826650] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 85.101162] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 86.935242] Key type lgssc registered [ 89.664693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.280393] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 138.404497] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 147.504511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.519707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.664672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.684560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.749722] Lustre: lustre-MDT0000: new disk, initializing [ 148.822116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.836050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.594245] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.120555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.162295] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.191485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.193593] Lustre: Skipped 1 previous similar message [ 160.242541] Lustre: lustre-MDT0001: new disk, initializing [ 160.279694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.294140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.307508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.234649] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 165.124871] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 170.337674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 170.532968] Lustre: lustre-OST0000: new disk, initializing [ 170.536160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 170.543345] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 170.581372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.851480] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.146734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.151548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.188286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.874323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.949273] Lustre: lustre-OST0001: new disk, initializing [ 181.953622] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 181.959467] Lustre: 9515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 182.003270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 184.681932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.695523] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.744344] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 185.461765] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.192173] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 197.012346] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 200.730704] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing check_logdir /tmp/testlogs/ [ 203.442413] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing yml_node [ 205.830198] Lustre: DEBUG MARKER: Client: 2.17.55.26 [ 207.044283] Lustre: DEBUG MARKER: MDS: 2.17.55.26 [ 208.307054] Lustre: DEBUG MARKER: OSS: 2.17.55.26 [ 209.115488] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:38:46 EDT 2026 [ 217.160423] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 217.840778] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 230.880623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.882579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 230.886930] Lustre: Skipped 3 previous similar messages [ 230.890215] Lustre: Skipped 2 previous similar messages [ 236.001367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 236.007510] Lustre: Skipped 3 previous similar messages [ 236.595712] Lustre: server umount lustre-MDT0000 complete [ 238.174189] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903557 with bad export cookie 1990179583823878014 [ 238.175751] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 238.180350] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 241.120318] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 241.126579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 241.130802] Lustre: Skipped 1 previous similar message [ 243.167807] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 243.173223] Lustre: Skipped 1 previous similar message [ 244.366315] Lustre: server umount lustre-MDT0001 complete [ 246.267859] Lustre: server umount lustre-OST0000 complete [ 248.136633] Lustre: server umount lustre-OST0001 complete [ 251.308329] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 255.998453] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 272.412372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.521059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 272.538819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 272.582659] Lustre: lustre-MDT0000: new disk, initializing [ 272.625955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.636374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 274.278130] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 279.666993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.744665] Lustre: 15053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 279.906706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 281.759644] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 286.619272] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 289.173892] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 292.329111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.462626] Lustre: lustre-OST0000: new disk, initializing [ 292.464618] Lustre: Skipped 1 previous similar message [ 292.466971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 292.470124] Lustre: Skipped 2 previous similar messages [ 292.472456] Lustre: 17022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.515757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 292.518558] Lustre: Skipped 1 previous similar message [ 294.080982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 294.084842] Lustre: Skipped 1 previous similar message [ 294.087096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 294.132488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 295.096863] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 299.716354] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 301.660602] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 301.770369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 303.756489] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 303.847641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 309.732417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 309.733885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.736015] LustreError: Skipped 1 previous similar message [ 309.736862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 309.748964] Lustre: Skipped 2 previous similar messages [ 310.956752] Lustre: server umount lustre-OST0000 complete [ 314.849616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.855619] Lustre: Skipped 1 previous similar message [ 318.700029] Lustre: server umount lustre-MDT0000 complete [ 319.969805] LustreError: 16002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.978752] LustreError: 16002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 320.323358] LustreError: 17021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903639 with bad export cookie 1990179583823879547 [ 320.327097] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.330488] LustreError: 17021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 320.478299] Lustre: server umount lustre-MDT0001 complete [ 322.158473] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:40:39 (1788903639) [ 325.575081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 325.850634] LustreError: 19283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.901692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.847666] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 331.234252] LustreError: 19284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.016955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.568613] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 338.347650] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 340.997236] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 344.359281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 347.498210] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 350.758376] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 362.977348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 362.977500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.979893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 362.979900] Lustre: Skipped 3 previous similar messages [ 362.988573] Lustre: Skipped 1 previous similar message [ 377.311155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.378837] Lustre: server umount lustre-OST0000 complete [ 383.456667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.457339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 383.457987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.457991] Lustre: Skipped 5 previous similar messages [ 383.462279] Lustre: Skipped 1 previous similar message [ 385.189770] Lustre: server umount lustre-MDT0000 complete [ 386.785493] LustreError: 19263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903705 with bad export cookie 1990179583823880429 [ 386.789948] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.791134] LustreError: 19263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 386.927693] Lustre: server umount lustre-MDT0001 complete [ 391.323267] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 392.563598] Key type lgssc unregistered [ 392.712662] LNet: 22840:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.716528] LNetError: 22840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 392.724331] LNet: Removed LNI 192.168.204.119@tcp [ 393.159309] Key type .llcrypt unregistered [ 393.161072] Key type ._llcrypt unregistered [ 403.574633] Key type ._llcrypt registered [ 403.576038] Key type .llcrypt registered [ 403.620953] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 403.992916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 404.029780] alg: No test for adler32 (adler32-zlib) [ 404.913582] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 405.020362] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 406.615229] Key type lgssc registered [ 407.084864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 410.407988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 410.417304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.618795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.274548] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 416.916900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.128510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.774597] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 421.874857] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.127185] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.093299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.253310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.611185] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 432.444159] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.453677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 446.944553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.950378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.953109] LustreError: Skipped 1 previous similar message [ 446.964094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.968093] Lustre: Skipped 1 previous similar message [ 449.553033] Lustre: server umount lustre-OST0000 complete [ 452.065955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.067208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 452.070840] Lustre: Skipped 2 previous similar messages [ 457.186751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.189015] Lustre: Skipped 2 previous similar messages [ 462.304177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 462.307044] Lustre: Skipped 1 previous similar message [ 465.375127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.459335] Lustre: server umount lustre-MDT0000 complete [ 466.402307] LustreError: 24228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.408022] LustreError: 24228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 466.931993] LustreError: 24211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903785 with bad export cookie 3552126141017940806 [ 466.934796] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 466.935685] LustreError: 24211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 467.056602] Lustre: server umount lustre-MDT0001 complete [ 471.557862] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 472.755951] Key type lgssc unregistered [ 472.913353] LNet: 28198:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.917038] LNetError: 28198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.925428] LNet: Removed LNI 192.168.204.119@tcp [ 473.291409] Key type .llcrypt unregistered [ 473.292837] Key type ._llcrypt unregistered [ 483.675930] Key type ._llcrypt registered [ 483.677517] Key type .llcrypt registered [ 483.728610] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 484.189978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 484.199779] alg: No test for adler32 (adler32-zlib) [ 485.102516] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 485.222548] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 486.824220] Key type lgssc registered [ 487.300290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 490.202771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 490.210600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.380813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.929336] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 496.112253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.263077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 497.777426] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 500.453888] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 502.521838] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 507.002892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 507.111935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.936358] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 511.269142] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.329912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 526.818181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.818819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.821847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.823219] LustreError: Skipped 1 previous similar message [ 526.825823] Lustre: Skipped 1 previous similar message [ 530.316910] Lustre: server umount lustre-OST0000 complete [ 531.936988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.937833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.940412] Lustre: Skipped 2 previous similar messages [ 531.945568] Lustre: Skipped 1 previous similar message [ 537.056491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 537.058261] Lustre: Skipped 1 previous similar message [ 537.691609] Lustre: server umount lustre-MDT0000 complete [ 538.927172] LustreError: 29567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903857 with bad export cookie 17111392651986896250 [ 538.931638] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.069370] Lustre: server umount lustre-MDT0001 complete [ 542.621326] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 543.650942] Key type lgssc unregistered [ 543.782392] LNet: 33482:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.785269] LNetError: 33482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 543.794365] LNet: Removed LNI 192.168.204.119@tcp [ 544.099536] Key type .llcrypt unregistered [ 544.101224] Key type ._llcrypt unregistered [ 552.554203] Key type ._llcrypt registered [ 552.555050] Key type .llcrypt registered [ 552.587717] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 552.880635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.908503] alg: No test for adler32 (adler32-zlib) [ 553.749911] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 553.824513] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 555.399151] Key type lgssc registered [ 555.739979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.283908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 558.289767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.416851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.686803] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 563.311018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.436109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.701305] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 567.043566] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 568.867985] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 571.067612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 571.174205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.051349] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 575.408317] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 584.485351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 588.767736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 588.769566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.773935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.792562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.793655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.795967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.798855] Lustre: server umount lustre-OST0000 complete [ 594.130719] Lustre: server umount lustre-MDT0000 complete [ 594.400057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 594.403047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.408481] LustreError: 36319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.413081] LustreError: 36319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 594.913789] LustreError: 34866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.309073] LustreError: 36711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903914 with bad export cookie 5280347183208363111 [ 595.311766] LustreError: 36711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 595.311995] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.417210] Lustre: server umount lustre-MDT0001 complete [ 599.057917] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 600.064951] Key type lgssc unregistered [ 600.179355] LNet: 38417:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.181521] LNetError: 38417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 600.192236] LNet: Removed LNI 192.168.204.119@tcp [ 600.492765] Key type .llcrypt unregistered [ 600.494398] Key type ._llcrypt unregistered [ 609.410953] Key type ._llcrypt registered [ 609.412393] Key type .llcrypt registered [ 609.452058] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 609.821712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 609.834840] alg: No test for adler32 (adler32-zlib) [ 610.696602] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 610.795062] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 612.375146] Key type lgssc registered [ 612.732306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.213545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 615.220085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.351403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.624170] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 620.285095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.393413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 621.720099] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 624.094371] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 625.947376] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 628.165938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 628.280336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.212036] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 632.571407] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 641.574703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 645.599705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 645.601593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.604854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.624583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 646.624757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 646.625762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.634910] Lustre: Skipped 1 previous similar message [ 648.007967] Lustre: lustre-OST0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 651.155759] Lustre: server umount lustre-OST0000 complete [ 663.875323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.876127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.879292] Lustre: Skipped 1 previous similar message [ 663.883021] Lustre: Skipped 1 previous similar message [ 667.936035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.937810] Lustre: Skipped 1 previous similar message [ 677.344102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.346851] Lustre: Skipped 3 previous similar messages [ 678.367077] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 678.423555] Lustre: server umount lustre-MDT0000 complete [ 679.595045] LustreError: 39788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788903998 with bad export cookie 16410908984186486681 [ 679.598528] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.599148] LustreError: 39788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 685.738856] Lustre: server umount lustre-MDT0001 complete [ 689.063448] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 689.992605] Key type lgssc unregistered [ 690.109325] LNet: 43534:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.111376] LNetError: 43534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.117253] LNet: Removed LNI 192.168.204.119@tcp [ 690.400414] Key type .llcrypt unregistered [ 690.401519] Key type ._llcrypt unregistered [ 698.941191] Key type ._llcrypt registered [ 698.942108] Key type .llcrypt registered [ 698.977216] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 699.285691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.305300] alg: No test for adler32 (adler32-zlib) [ 700.153888] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 700.235435] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 701.815122] Key type lgssc registered [ 702.163066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 704.550072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 704.555993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.676728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.892444] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 709.496612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.598555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 710.778303] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 713.036474] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 714.905137] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 717.181164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.282085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.158760] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 721.494987] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 730.601601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 735.199749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 735.201587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.205471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.713414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.714318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.718645] Lustre: Skipped 1 previous similar message [ 737.094936] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.19@tcp (stopping) [ 737.096434] Lustre: Skipped 1 previous similar message [ 740.439184] Lustre: server umount lustre-MDT0000 complete [ 740.832086] LustreError: 47413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.836836] LustreError: 47413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 741.652149] LustreError: 44902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904060 with bad export cookie 2924471418085622901 [ 741.655357] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.656383] LustreError: 44902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 742.209707] LustreError: 44916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.215258] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.19@tcp (stopping) [ 745.951893] LustreError: 44918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 745.952231] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.976280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 746.977933] Lustre: Skipped 1 previous similar message [ 747.811780] Lustre: server umount lustre-MDT0001 complete [ 765.713656] Lustre: server umount lustre-OST0000 complete [ 770.813987] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 771.708715] Key type lgssc unregistered [ 771.833332] LNet: 48800:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.835156] LNetError: 48800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.844219] LNet: Removed LNI 192.168.204.119@tcp [ 772.108943] Key type .llcrypt unregistered [ 772.109816] Key type ._llcrypt unregistered [ 781.467092] Key type ._llcrypt registered [ 781.467989] Key type .llcrypt registered [ 781.502611] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 781.846893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 781.892084] alg: No test for adler32 (adler32-zlib) [ 782.759892] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 782.856785] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 784.447211] Key type lgssc registered [ 784.835104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 787.482807] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 787.489338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.911316] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 808.031287] LustreError: 50163:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 834.655220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.743172] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 839.470427] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1005.125124] Lustre: server umount lustre-OST0000 complete [ 1024.193955] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1028.322743] Key type lgssc unregistered [ 1028.787036] LNet: 51841:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.792662] LNetError: 51841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.867030] LNet: Removed LNI 192.168.204.119@tcp [ 1030.914763] Key type .llcrypt unregistered [ 1030.917100] Key type ._llcrypt unregistered [ 1055.784463] Key type ._llcrypt registered [ 1055.791135] Key type .llcrypt registered [ 1055.961316] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1056.823790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.867872] alg: No test for adler32 (adler32-zlib) [ 1057.831262] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1057.988899] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1059.663574] Key type lgssc registered [ 1060.886732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1069.765350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1069.786833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.455260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.237501] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1081.840147] LustreError: 53248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.889888] LustreError: 53248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1085.526363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.879895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.522746] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1098.861429] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1104.939711] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1112.636437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1113.122561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.219234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1119.897696] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1127.608152] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1130.729557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1130.729848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.896826] Lustre: server umount lustre-OST0000 complete [ 1141.219776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.222465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.234840] Lustre: Skipped 1 previous similar message [ 1141.247034] Lustre: Skipped 2 previous similar messages [ 1146.139632] Lustre: server umount lustre-MDT0000 complete [ 1146.360109] LustreError: 53248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.387708] LustreError: 53248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1149.914936] LustreError: 53229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904468 with bad export cookie 548224421192645610 [ 1149.922882] LustreError: 53229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1149.928437] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.355559] Lustre: server umount lustre-MDT0001 complete [ 1161.715266] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1164.679661] Key type lgssc unregistered [ 1165.014030] LNet: 56779:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.025770] LNetError: 56779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.048953] LNet: Removed LNI 192.168.204.119@tcp [ 1166.437302] Key type .llcrypt unregistered [ 1166.439707] Key type ._llcrypt unregistered [ 1193.445548] Key type ._llcrypt registered [ 1193.453561] Key type .llcrypt registered [ 1193.642349] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1194.830652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1194.859889] alg: No test for adler32 (adler32-zlib) [ 1196.034270] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1196.294922] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1197.951162] Key type lgssc registered [ 1199.323551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1206.992582] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.031238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.561168] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1227.680379] LustreError: 58171:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 1254.305978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.406546] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1268.918451] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1279.380124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.346087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.428628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1285.602954] LustreError: 58197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.610963] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1285.617513] LustreError: 58197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1288.694893] LustreError: 59208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.725055] LustreError: 59208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1293.794480] LustreError: 59209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.823067] LustreError: 59209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1296.744554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.215295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.834474] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1311.024926] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1316.372878] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1321.445097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1321.451829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.471840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.981601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1323.008639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.008891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1324.886465] Lustre: server umount lustre-OST0000 complete [ 1347.552785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.557322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.571081] Lustre: Skipped 1 previous similar message [ 1347.576778] Lustre: Skipped 1 previous similar message [ 1352.673883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.684109] Lustre: Skipped 2 previous similar messages [ 1357.798409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.805296] Lustre: Skipped 1 previous similar message [ 1361.376526] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1361.799475] Lustre: server umount lustre-MDT0000 complete [ 1362.920193] LustreError: 59207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.939393] LustreError: 59207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1366.218335] LustreError: 58179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904685 with bad export cookie 12682747231018683950 [ 1366.226606] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.235576] LustreError: 58179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1372.668765] Lustre: server umount lustre-MDT0001 complete [ 1386.682628] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1391.062651] Key type lgssc unregistered [ 1391.487972] LNet: 61919:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.496514] LNetError: 61919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.527992] LNet: Removed LNI 192.168.204.119@tcp [ 1392.894799] Key type .llcrypt unregistered [ 1392.901840] Key type ._llcrypt unregistered [ 1419.395643] Key type ._llcrypt registered [ 1419.397548] Key type .llcrypt registered [ 1419.504810] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1420.175419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.219697] alg: No test for adler32 (adler32-zlib) [ 1421.266247] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1421.472104] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1423.095193] Key type lgssc registered [ 1423.947065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1429.664168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1429.677603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.064441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.964213] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1440.179920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.225970] LustreError: 63332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.245451] LustreError: 63332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1440.435972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1443.441850] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1448.331562] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1451.944816] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1456.570414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1456.824868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.870833] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1465.422758] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1475.242782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1480.676750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1480.678365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.681350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.694840] Lustre: Skipped 1 previous similar message [ 1485.097263] Lustre: server umount lustre-OST0000 complete [ 1490.919202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.933790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1490.940269] Lustre: Skipped 2 previous similar messages [ 1493.449621] Lustre: server umount lustre-MDT0000 complete [ 1495.516669] LustreError: 63313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904814 with bad export cookie 11536621434043148807 [ 1495.519680] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.521939] LustreError: 63313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1495.681705] Lustre: server umount lustre-MDT0001 complete [ 1500.865432] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1502.250291] Key type lgssc unregistered [ 1502.414850] LNet: 66882:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.419561] LNetError: 66882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.430731] LNet: Removed LNI 192.168.204.119@tcp [ 1502.873839] Key type .llcrypt unregistered [ 1502.880304] Key type ._llcrypt unregistered [ 1516.312441] Key type ._llcrypt registered [ 1516.314051] Key type .llcrypt registered [ 1516.374556] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1516.891315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1516.960978] alg: No test for adler32 (adler32-zlib) [ 1517.889361] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1518.015320] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1519.623642] Key type lgssc registered [ 1520.291705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.810381] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:00:42 (1788904842) [ 1528.328574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1528.335433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1529.570500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.586397] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1536.288442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.498085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.779625] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1542.649849] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1545.780988] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1549.117917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1549.291168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.273496] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1554.419057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1555.901561] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1564.357756] Lustre: Failing over lustre-MDT0000 [ 1564.525093] Lustre: server umount lustre-MDT0000 complete [ 1566.688829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1566.694709] Lustre: Skipped 2 previous similar messages [ 1566.694835] LustreError: 70122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.706572] LustreError: 70122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1568.597930] LustreError: 68559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1569.190171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.254987] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.328977] Lustre: *** cfs_fail_loc=135, val=0*** [ 1569.378926] LustreError: 71660:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1569.383703] LustreError: 71660:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.119@tcp: cfg command failed: rc = -2 [ 1569.387482] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1569.387482] [ 1569.393075] LustreError: MGC192.168.204.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1569.400130] LustreError: 71649:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1569.404663] LustreError: 71649:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 1569.408735] LustreError: 71649:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 1569.414233] LustreError: 71649:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1569.427548] Lustre: server umount lustre-MDT0000 complete [ 1569.429422] LustreError: 71649:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1571.809111] LustreError: 68564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1571.815969] LustreError: 68564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1572.790522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.855503] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1572.996114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.023088] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1573.702681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1575.191660] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1578.471100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1578.487779] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1578.525348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1582.049316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1582.056766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1582.069283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1583.589459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.591610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1583.592068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.266336] Lustre: server umount lustre-OST0000 complete [ 1588.704794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1588.705185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.705995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1588.706018] Lustre: Skipped 1 previous similar message [ 1588.719502] Lustre: Skipped 1 previous similar message [ 1593.825511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.828774] Lustre: Skipped 2 previous similar messages [ 1593.994660] Lustre: server umount lustre-MDT0000 complete [ 1595.803916] LustreError: 70417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788904914 with bad export cookie 11260298710988467843 [ 1595.804356] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.815393] LustreError: 70417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1595.958192] Lustre: server umount lustre-MDT0001 complete [ 1601.080111] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1602.495885] Key type lgssc unregistered [ 1602.667654] LNet: 73434:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.671119] LNetError: 73434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.682540] LNet: Removed LNI 192.168.204.119@tcp [ 1603.140228] Key type .llcrypt unregistered [ 1603.144175] Key type ._llcrypt unregistered [ 1614.511589] Key type ._llcrypt registered [ 1614.513195] Key type .llcrypt registered [ 1614.561949] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1615.155132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1615.165410] alg: No test for adler32 (adler32-zlib) [ 1616.059882] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1616.173380] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1617.775173] Key type lgssc registered [ 1618.220697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1621.634532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1621.641456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.831484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.444867] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1627.959794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.144928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1629.799300] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1632.887351] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1635.053214] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1637.678600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.827677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.139584] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1642.979512] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1648.105840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1656.464124] Lustre: Failing over lustre-MDT0000 [ 1656.641745] Lustre: server umount lustre-MDT0000 complete [ 1657.315383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.317239] LustreError: 75554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.319864] Lustre: Skipped 1 previous similar message [ 1657.326083] LustreError: 75554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1659.966631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.017020] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.086541] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1660.148634] LustreError: 77933:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1660.151949] LustreError: 77933:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.119@tcp: cfg command failed: rc = -9 [ 1660.155323] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1660.155323] [ 1660.159255] LustreError: MGC192.168.204.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1660.163260] LustreError: 77921:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1660.165857] LustreError: 77921:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -9 [ 1660.168534] LustreError: 77921:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 1660.170730] LustreError: 77921:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1660.183545] Lustre: server umount lustre-MDT0000 complete [ 1660.185149] LustreError: 77921:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1660.747223] LustreError: 74821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.754709] LustreError: 74821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1662.433110] LustreError: 77301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.438267] LustreError: 77301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1662.585341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.622686] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1662.711894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.740409] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1664.137268] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1665.858358] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1668.071645] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1668.083465] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1668.103376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1672.671898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1672.675161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.679621] Lustre: Skipped 1 previous similar message [ 1672.681993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.187163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.188024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.236387] Lustre: server umount lustre-OST0000 complete [ 1678.305053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1678.305313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.306146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.306151] Lustre: Skipped 1 previous similar message [ 1678.318988] Lustre: Skipped 2 previous similar messages [ 1683.424647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.427371] Lustre: Skipped 2 previous similar messages [ 1688.544760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1688.547391] Lustre: Skipped 1 previous similar message [ 1691.103716] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1691.176111] Lustre: server umount lustre-MDT0000 complete [ 1692.654489] LustreError: 74806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905011 with bad export cookie 1986209908876646340 [ 1692.656980] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.658776] LustreError: 74806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1692.779393] Lustre: server umount lustre-MDT0001 complete [ 1696.525668] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1697.574413] Key type lgssc unregistered [ 1697.693336] LNet: 79709:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.695818] LNetError: 79709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.708336] LNet: Removed LNI 192.168.204.119@tcp [ 1698.016585] Key type .llcrypt unregistered [ 1698.017566] Key type ._llcrypt unregistered [ 1702.192955] Key type ._llcrypt registered [ 1702.194417] Key type .llcrypt registered [ 1702.233558] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1702.355647] Key type .llcrypt unregistered [ 1702.357320] Key type ._llcrypt unregistered [ 1711.305413] Key type ._llcrypt registered [ 1711.306829] Key type .llcrypt registered [ 1711.349695] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1711.639798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.659348] alg: No test for adler32 (adler32-zlib) [ 1712.520407] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1712.614950] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1714.199140] Key type lgssc registered [ 1714.586895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.481867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.493721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.715406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.280303] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1723.397914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.532750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1725.035898] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1727.734563] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1729.996490] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1732.739117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1732.900636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.283796] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1738.052654] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1743.146694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1748.450228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1748.450627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.454386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1748.458541] Lustre: Skipped 1 previous similar message [ 1752.470781] Lustre: server umount lustre-OST0000 complete [ 1754.592257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.596058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.602200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.604517] Lustre: Skipped 1 previous similar message [ 1758.688526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.689279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.100852] Lustre: server umount lustre-MDT0000 complete [ 1761.467399] LustreError: 81567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905080 with bad export cookie 16182242420050440540 [ 1761.472361] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.475221] LustreError: 81567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1761.586149] Lustre: server umount lustre-MDT0001 complete [ 1765.221915] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1766.362803] Key type lgssc unregistered [ 1766.497405] LNet: 85114:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.500316] LNetError: 85114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.506318] LNet: Removed LNI 192.168.204.119@tcp [ 1766.833671] Key type .llcrypt unregistered [ 1766.834816] Key type ._llcrypt unregistered [ 1775.598620] Key type ._llcrypt registered [ 1775.599495] Key type .llcrypt registered [ 1775.636349] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1775.964166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.982146] alg: No test for adler32 (adler32-zlib) [ 1776.842893] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1776.947932] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1778.535239] Key type lgssc registered [ 1778.946989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1781.761968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1781.768678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.921973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.412408] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1787.333684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.467118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1788.908756] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1791.418576] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1793.298126] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1795.633326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1795.765264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.820423] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1800.361330] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1809.061953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 1813.471918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1813.474530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.479232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1814.498163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1814.498814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.498850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.190840] Lustre: server umount lustre-OST0000 complete [ 1818.079952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1818.083284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.088397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.091307] Lustre: Skipped 1 previous similar message [ 1818.647670] Lustre: server umount lustre-MDT0000 complete [ 1819.616607] LustreError: 87988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.625547] LustreError: 87988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1819.982121] LustreError: 86485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905138 with bad export cookie 7913563373869153147 [ 1819.984431] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1819.986698] LustreError: 86485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1820.109590] Lustre: server umount lustre-MDT0001 complete [ 1823.964536] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1825.124847] Key type lgssc unregistered [ 1825.266888] LNet: 90051:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.270202] LNetError: 90051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.277752] LNet: Removed LNI 192.168.204.119@tcp [ 1825.618459] Key type .llcrypt unregistered [ 1825.620263] Key type ._llcrypt unregistered [ 1834.802397] Key type ._llcrypt registered [ 1834.803773] Key type .llcrypt registered [ 1834.840645] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1835.140890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.148951] alg: No test for adler32 (adler32-zlib) [ 1836.022677] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1836.111682] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1837.695238] Key type lgssc registered [ 1838.070057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.774655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.781232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.944481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.342379] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1846.324769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.453505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.946710] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1850.595097] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.534096] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.936514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.050730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1857.071913] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1859.502305] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1866.347559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 1876.963588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.964960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.967096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.973982] Lustre: Skipped 1 previous similar message [ 1879.382108] Lustre: server umount lustre-OST0000 complete [ 1882.081994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.082667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.086068] Lustre: Skipped 1 previous similar message [ 1882.088143] Lustre: Skipped 1 previous similar message [ 1886.878480] Lustre: server umount lustre-MDT0000 complete [ 1887.200960] LustreError: 92832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.207938] LustreError: 92832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1888.217761] LustreError: 91423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905207 with bad export cookie 8641713468215920580 [ 1888.221388] LustreError: 91423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1888.221511] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.336196] Lustre: server umount lustre-MDT0001 complete [ 1892.002133] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 1893.073497] Key type lgssc unregistered [ 1893.200460] LNet: 94995:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.203538] LNetError: 94995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.209307] LNet: Removed LNI 192.168.204.119@tcp [ 1893.548683] Key type .llcrypt unregistered [ 1893.550160] Key type ._llcrypt unregistered [ 1902.899272] Key type ._llcrypt registered [ 1902.900754] Key type .llcrypt registered [ 1902.947270] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 1903.389023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.413561] alg: No test for adler32 (adler32-zlib) [ 1904.289693] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 1904.394438] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 1905.991212] Key type lgssc registered [ 1906.402924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.271717] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.281371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1924.767373] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1929.887287] LustreError: 96359:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 1956.511291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.591315] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1961.208527] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1963.252866] Lustre: server umount lustre-OST0000 complete [ 1966.252043] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:08:04 (1788905284) [ 1968.733584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.912541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.408578] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1973.479330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.633581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1975.137298] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1977.886232] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.969673] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.516530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1982.664133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.828299] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 1987.460826] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1993.962863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2004.452604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2004.455297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.456268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.460539] Lustre: Skipped 1 previous similar message [ 2007.700471] Lustre: server umount lustre-OST0000 complete [ 2009.568501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.568582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2009.574757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.577262] LustreError: Skipped 1 previous similar message [ 2009.580926] Lustre: Skipped 2 previous similar messages [ 2014.688651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.690612] Lustre: Skipped 1 previous similar message [ 2019.808642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.811610] Lustre: Skipped 1 previous similar message [ 2023.391093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2023.444860] Lustre: server umount lustre-MDT0000 complete [ 2024.749463] LustreError: 97975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905343 with bad export cookie 9472544954943786742 [ 2024.752438] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.753868] LustreError: 97975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2024.871910] Lustre: server umount lustre-MDT0001 complete [ 2028.705798] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2029.801751] Key type lgssc unregistered [ 2029.928327] LNet: 101705:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.931674] LNetError: 101705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.941326] LNet: Removed LNI 192.168.204.119@tcp [ 2030.269262] Key type .llcrypt unregistered [ 2030.270693] Key type ._llcrypt unregistered [ 2045.224495] Key type ._llcrypt registered [ 2045.225529] Key type .llcrypt registered [ 2045.263126] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2045.569075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.592027] alg: No test for adler32 (adler32-zlib) [ 2046.456627] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2046.550570] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2048.135191] Key type lgssc registered [ 2048.573195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2051.255588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2051.261799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.404345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.693663] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2056.602786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.734441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.205171] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2060.751607] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2062.592054] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2064.886269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2065.002926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.911306] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2069.204500] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2077.284942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2082.785366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2082.785813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.788410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2082.792629] Lustre: Skipped 1 previous similar message [ 2084.237401] Lustre: server umount lustre-OST0000 complete [ 2087.391828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2087.394316] LustreError: Skipped 1 previous similar message [ 2087.395660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.400807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2087.402600] Lustre: Skipped 1 previous similar message [ 2087.568113] Lustre: server umount lustre-MDT0000 complete [ 2087.905870] LustreError: 103859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.912095] LustreError: 103859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2088.820517] LustreError: 103844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905407 with bad export cookie 11314615712470999532 [ 2088.823723] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2088.824620] LustreError: 103844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2088.928222] Lustre: server umount lustre-MDT0001 complete [ 2092.440374] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2093.501491] Key type lgssc unregistered [ 2093.623445] LNet: 107412:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.626760] LNetError: 107412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.637308] LNet: Removed LNI 192.168.204.119@tcp [ 2093.953866] Key type .llcrypt unregistered [ 2093.955096] Key type ._llcrypt unregistered [ 2101.285714] Key type ._llcrypt registered [ 2101.286929] Key type .llcrypt registered [ 2101.327679] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2101.698480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.746455] alg: No test for adler32 (adler32-zlib) [ 2102.607967] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2102.700663] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2104.287138] Key type lgssc registered [ 2104.656648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.273325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2107.280107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.719365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2127.839253] LustreError: 108632:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 2154.463199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2156.359015] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2158.831828] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2161.253534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.355052] LustreError: MGC192.168.204.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2161.359787] LustreError: 109655:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2161.362912] LustreError: 109655:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 2161.366193] LustreError: 109655:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 2161.398449] Lustre: server umount lustre-MDT0000 complete [ 2161.400693] LustreError: 109655:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2173.442672] Lustre: server umount lustre-OST0000 complete [ 2176.317534] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 2179.253485] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2190.671350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.757092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2190.768081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2190.807356] Lustre: lustre-MDT0000: new disk, initializing [ 2190.835267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.843866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2192.125664] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2196.347853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.387667] Lustre: 112439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2196.410870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2196.412293] Lustre: Skipped 1 previous similar message [ 2196.450108] Lustre: lustre-MDT0001: new disk, initializing [ 2196.481170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.491023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2196.493660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2197.797332] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2201.502305] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2203.369276] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.676223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.789425] Lustre: lustre-OST0000: new disk, initializing [ 2205.791218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2205.794047] Lustre: 114409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2205.827092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.040792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2206.045643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2206.082220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2207.802588] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2211.560112] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2212.981659] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2213.054406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2214.468900] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2214.545344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2216.419860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.422136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.422219] LustreError: Skipped 1 previous similar message [ 2216.428229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.430761] Lustre: Skipped 1 previous similar message [ 2221.333310] Lustre: server umount lustre-OST0000 complete [ 2226.656568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.657180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2226.660295] Lustre: Skipped 2 previous similar messages [ 2228.825690] Lustre: server umount lustre-MDT0000 complete [ 2230.111595] LustreError: 113235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905548 with bad export cookie 7993303295775366040 [ 2230.114837] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.116873] LustreError: 113235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2230.229954] Lustre: server umount lustre-MDT0001 complete [ 2233.008796] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:12:31 (1788905551) [ 2234.133338] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2239.712212] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 2242.738181] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2264.293476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.496770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2264.516425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2264.586454] Lustre: lustre-MDT0000: new disk, initializing [ 2264.654660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.672790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2268.151632] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2281.553458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.734622] Lustre: 119299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2281.841761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2281.853695] Lustre: Skipped 1 previous similar message [ 2281.978994] Lustre: lustre-MDT0001: new disk, initializing [ 2282.117780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2282.172921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2282.195194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2288.046200] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2299.549322] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2306.722936] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2313.763917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2314.140836] Lustre: lustre-OST0000: new disk, initializing [ 2314.147894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2314.163349] Lustre: 121267:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2314.291429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.285272] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2321.425502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2321.431829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2321.498152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2324.096764] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2326.715237] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2326.842864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2329.165510] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2329.278367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2331.618734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2331.619294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.621159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2331.621165] Lustre: Skipped 1 previous similar message [ 2331.622876] LustreError: Skipped 1 previous similar message [ 2331.641435] Lustre: Skipped 1 previous similar message [ 2336.506138] Lustre: server umount lustre-OST0000 complete [ 2341.856373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.858220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2341.864018] Lustre: Skipped 1 previous similar message [ 2341.866552] Lustre: Skipped 1 previous similar message [ 2344.893511] Lustre: server umount lustre-MDT0000 complete [ 2346.992327] LustreError: 119291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905665 with bad export cookie 7993303295775366908 [ 2346.995670] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.998904] LustreError: 119291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2347.198266] Lustre: server umount lustre-MDT0001 complete [ 2351.870437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.099365] LustreError: 123482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.108271] LustreError: 123482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2352.136244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.269895] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2357.221176] LustreError: 123483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.114498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.696411] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2365.884807] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2368.988203] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2372.675753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2376.082296] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2380.403581] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2388.330597] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2392.544896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2392.548849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.559982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.564317] Lustre: Skipped 1 previous similar message [ 2394.595920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.507592] Lustre: server umount lustre-OST0000 complete [ 2404.554866] Lustre: server umount lustre-MDT0000 complete [ 2404.832918] LustreError: 123483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.843641] LustreError: 123483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2406.537488] LustreError: 124220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905725 with bad export cookie 7993303295775367790 [ 2406.540236] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2406.544923] LustreError: 124220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2406.714142] Lustre: server umount lustre-MDT0001 complete [ 2412.324659] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 2413.899698] Key type lgssc unregistered [ 2414.078491] LNet: 127107:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.082641] LNetError: 127107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.097740] LNet: Removed LNI 192.168.204.119@tcp [ 2414.581257] Key type .llcrypt unregistered [ 2414.583586] Key type ._llcrypt unregistered [ 2424.676302] Key type ._llcrypt registered [ 2424.678219] Key type .llcrypt registered [ 2424.754305] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 2434.355208] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 2434.787490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2434.800082] alg: No test for adler32 (adler32-zlib) [ 2435.718869] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 2435.857735] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 2437.479198] Key type lgssc registered [ 2438.231069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.192357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.212421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.387867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2459.415672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2459.504334] Lustre: lustre-MDT0000: new disk, initializing [ 2459.581671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.592639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2462.206889] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2469.388342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.481928] Lustre: 130235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2469.541478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2469.544816] Lustre: Skipped 1 previous similar message [ 2469.613506] Lustre: lustre-MDT0001: new disk, initializing [ 2469.677233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.700865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2469.705911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2472.337957] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2478.709500] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2482.366350] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2486.759818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2486.950551] Lustre: lustre-OST0000: new disk, initializing [ 2486.953977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2486.960804] Lustre: 132205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2487.028238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.046702] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2493.507972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2493.519633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2493.593612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2497.662139] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2500.725548] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2500.889600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2503.699738] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2503.820449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2509.286544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2509.290849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.299621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.304376] Lustre: Skipped 1 previous similar message [ 2511.079529] Lustre: server umount lustre-OST0000 complete [ 2514.401637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.408152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.414058] Lustre: Skipped 2 previous similar messages [ 2514.421372] Lustre: Skipped 1 previous similar message [ 2519.533838] Lustre: server umount lustre-MDT0000 complete [ 2521.670563] LustreError: 133745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905840 with bad export cookie 12787012716348481099 [ 2521.678499] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.679244] LustreError: 133745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2521.894220] Lustre: server umount lustre-MDT0001 complete [ 2527.127078] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:17:24 (1788905844) [ 2531.084931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.318159] LustreError: 134614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.328888] LustreError: 134614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2531.356649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.640639] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2536.416944] LustreError: 134615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.351480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.542341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2540.945899] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2545.257087] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2549.149399] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2551.329497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.331980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.336562] Lustre: Skipped 1 previous similar message [ 2551.341567] Lustre: Skipped 1 previous similar message [ 2556.387436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.397618] Lustre: Skipped 1 previous similar message [ 2557.556501] Lustre: server umount lustre-MDT0000 complete [ 2561.509585] LustreError: 134614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.552164] LustreError: 134614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2562.730019] LustreError: 134595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788905881 with bad export cookie 12787012716348481988 [ 2562.760242] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.761919] LustreError: 134595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2563.488503] Lustre: server umount lustre-MDT0001 complete [ 2576.356769] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:18:13 (1788905893) [ 2586.128946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.695938] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2606.817439] LustreError: 137029:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 2632.615233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.424855] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2649.659367] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2652.484725] Lustre: server umount lustre-OST0000 complete [ 2662.505193] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:19:39 (1788905979) [ 2669.889970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.444409] LustreError: 138477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.518180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.533726] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2675.684712] LustreError: 138478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.401791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.375629] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2697.273832] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2703.203158] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2710.216520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.501983] LustreError: 140297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.521181] LustreError: 140297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2710.686591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.691050] Lustre: Skipped 1 previous similar message [ 2718.015582] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2726.084353] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.097109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2736.113933] LustreError: Skipped 1 previous similar message [ 2736.120534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.139286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.517499] Lustre: server umount lustre-MDT0000 complete [ 2741.731254] LustreError: 138477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.753857] LustreError: 138477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2744.805269] LustreError: 138458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906063 with bad export cookie 12787012716348482639 [ 2744.821131] LustreError: 138458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2744.832333] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.171204] Lustre: server umount lustre-MDT0001 complete [ 2758.397736] Lustre: server umount lustre-OST0000 complete [ 2767.337961] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:21:24 (1788906084) [ 2774.730750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.106439] LustreError: 142138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.193201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.114606] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2790.788308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.705578] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2804.218479] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2811.559632] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2819.625668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.241505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2826.832851] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2835.429663] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2840.884640] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2841.155351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2846.858105] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2847.140635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2851.297739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.303976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2851.331862] Lustre: Skipped 2 previous similar messages [ 2851.333811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2851.333824] Lustre: Skipped 2 previous similar messages [ 2855.808766] Lustre: server umount lustre-OST0000 complete [ 2861.536858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.549090] Lustre: Skipped 1 previous similar message [ 2866.424325] Lustre: server umount lustre-MDT0000 complete [ 2866.668198] LustreError: 142139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.686546] LustreError: 142139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2870.240492] LustreError: 142119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906189 with bad export cookie 12787012716348483710 [ 2870.248424] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.250229] LustreError: 142119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2870.506526] Lustre: server umount lustre-MDT0001 complete [ 2879.548672] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:23:16 (1788906196) [ 2886.731708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2902.303823] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2907.423445] LustreError: 146298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 2933.151425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.158849] Lustre: Skipped 2 previous similar messages [ 2938.732656] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2947.077911] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2953.918387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.028431] LustreError: 128462:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000678be450 x1875803640025600/t0(0) o250->MGC192.168.204.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2954.608475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2959.383662] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2968.863154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.298345] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 2982.154298] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2988.046626] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2993.160744] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2993.387376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2998.282488] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2998.458417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3005.408718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3005.413600] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.437996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.438053] Lustre: Skipped 3 previous similar messages [ 3005.472878] Lustre: Skipped 2 previous similar messages [ 3015.137827] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3015.391124] Lustre: server umount lustre-MDT0000 complete [ 3015.656466] LustreError: 147336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.691766] LustreError: 147336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3019.378865] LustreError: 148080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906338 with bad export cookie 12787012716348484732 [ 3019.385206] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3019.390085] LustreError: 148080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3019.722459] Lustre: server umount lustre-MDT0001 complete [ 3033.918265] Lustre: server umount lustre-OST0000 complete [ 3043.548479] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:26:00 (1788906360) [ 3051.030943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.783736] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3071.905162] LustreError: 150495:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 3103.454996] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3115.020611] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3122.997340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.057465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3130.634786] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3142.205029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.610462] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3157.587450] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3165.637744] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3173.773491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.092931] Lustre: lustre-OST0001: new disk, initializing [ 3174.102879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3174.111404] Lustre: 153369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3182.047598] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3182.680071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3182.688783] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3182.875516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3193.445363] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3198.440194] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3198.661403] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3204.056495] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3204.490517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3208.684753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.693663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.705632] Lustre: Skipped 2 previous similar messages [ 3208.710645] Lustre: Skipped 5 previous similar messages [ 3213.134482] Lustre: server umount lustre-OST0000 complete [ 3223.757966] Lustre: server umount lustre-OST0001 complete [ 3234.526600] Lustre: server umount lustre-MDT0000 complete [ 3239.111623] LustreError: 150502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906557 with bad export cookie 12787012716348485684 [ 3239.113466] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.119513] LustreError: 150502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3239.435861] Lustre: server umount lustre-MDT0001 complete [ 3267.186919] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:29:44 (1788906584) [ 3268.971564] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3270.898238] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3272.684860] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:29:49 (1788906589) [ 3279.747134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.092522] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3280.144364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3280.351101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.360447] Lustre: Skipped 6 previous similar messages [ 3285.006607] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3294.602924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3294.765755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3294.785544] Lustre: 156933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3301.155425] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3310.342421] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3316.147349] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3323.263032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3323.567160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3330.212736] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3333.191655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3337.483627] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3342.268758] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3342.433764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3346.532745] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3346.717390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3348.453355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3348.458058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.465311] LustreError: Skipped 1 previous similar message [ 3348.467626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3348.489919] Lustre: Skipped 5 previous similar messages [ 3348.513345] Lustre: Skipped 8 previous similar messages [ 3354.380481] Lustre: server umount lustre-OST0000 complete [ 3375.993422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.427688] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3378.446535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3378.457602] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3378.462775] Lustre: Skipped 1 previous similar message [ 3381.650171] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3390.170248] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3395.450628] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3395.672921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3400.700802] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3400.925975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3406.109916] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff931042bb5000.ost_server_uuid 50 [ 3407.593269] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff931042bb5000.ost_server_uuid in FULL state after 0 sec [ 3407.746431] LustreError: 157856:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.19@tcp) returned error from blocking AST (req@00000000c89135db x1875803640166272 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004c18da66/0xb1749851285ab98e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.19@tcp remote: 0xa8c1e353128d94c2 expref: 5 pid: 156940 timeout: 3507 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3407.792396] LustreError: lustre-MDT0000: A client on nid 192.168.204.19@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3407.802134] LustreError: 156930:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.19@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004c18da66/0xb1749851285ab98e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.19@tcp remote: 0xa8c1e353128d94c2 expref: 6 pid: 156940 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3413.992412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3418.633853] Lustre: server umount lustre-OST0000 complete [ 3428.751642] Lustre: server umount lustre-MDT0000 complete [ 3429.868946] LustreError: 156945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.896367] LustreError: 156945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3432.794372] LustreError: 156926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906751 with bad export cookie 12787012716348486867 [ 3432.798449] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.804695] LustreError: 156926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3447.264229] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.635090] Lustre: server umount lustre-MDT0001 complete [ 3457.987950] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3460.774221] Key type lgssc unregistered [ 3461.107824] LNet: 163048:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3461.118279] LNetError: 163048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3461.131141] LNet: Removed LNI 192.168.204.119@tcp [ 3462.076314] Key type .llcrypt unregistered [ 3462.078674] Key type ._llcrypt unregistered [ 3484.473532] Key type ._llcrypt registered [ 3484.475657] Key type .llcrypt registered [ 3484.554568] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3485.451745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3485.598586] alg: No test for adler32 (adler32-zlib) [ 3486.613455] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3486.810198] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3488.455300] Key type lgssc registered [ 3489.514897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3496.456403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3496.474471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.898819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.875364] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3507.168602] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3507.183399] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3509.328589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.759882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3514.671397] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3523.434511] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3530.222300] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3537.740416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3538.077523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.205818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3543.954623] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3551.704914] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3560.801260] Lustre: Failing over lustre-MDT0000 [ 3560.932821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3560.941503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.965822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3561.255765] Lustre: server umount lustre-MDT0000 complete [ 3565.024736] LustreError: 164459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.025956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.145836] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.173875] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3573.797676] LustreError: 166305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.19@tcp arrived at 1788906892 with bad export cookie 14596785077554407205 [ 3573.820283] LustreError: 166305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3575.268357] LustreError: 164456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.280057] LustreError: 164456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3580.385983] Lustre: 163771:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788906883/real 1788906883] req@00000000ac998393 x1875804741987840/t0(0) o400->MGC192.168.204.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788906899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3580.394152] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.466351] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.473928] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3581.311103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.149846] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3591.189592] LustreError: 164460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3591.428458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.504573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3596.036775] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3596.779865] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3596.792813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3598.160203] Lustre: lustre-MDT0000: Denying connection for new client b60b15d0-e63e-4370-8134-1f8db4a5da1b (at 192.168.204.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 3605.985328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3605.989667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3605.997264] Lustre: Skipped 1 previous similar message [ 3606.002308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.010148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.134355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.138220] Lustre: Skipped 2 previous similar messages [ 3617.250224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.256386] Lustre: Skipped 1 previous similar message [ 3620.319137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3620.444270] Lustre: server umount lustre-OST0000 complete [ 3623.720097] LustreError: 168193:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3623.725289] Lustre: 167542:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3623.731210] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3623.741725] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3623.749692] LustreError: 167542:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008bf9e859 x1875804742008192/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3623.761407] LustreError: 167542:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3623.767762] LustreError: 167542:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3623.773315] Lustre: lustre-MDT0000: Recovery over after 0:27, of 2 clients 0 recovered and 2 were evicted. [ 3624.017904] Lustre: server umount lustre-MDT0000 complete [ 3627.318028] LustreError: 164440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788906946 with bad export cookie 14596785077554407310 [ 3627.322876] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.331320] LustreError: 164440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3627.332845] LustreError: 163769:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3633.660344] Lustre: server umount lustre-MDT0001 complete [ 3641.729497] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3644.027935] Key type lgssc unregistered [ 3644.290379] LNet: 168946:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3644.301185] LNetError: 168946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3644.339536] LNet: Removed LNI 192.168.204.119@tcp [ 3645.106108] Key type .llcrypt unregistered [ 3645.107826] Key type ._llcrypt unregistered [ 3665.067301] Key type ._llcrypt registered [ 3665.069338] Key type .llcrypt registered [ 3665.196821] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3666.019687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3666.103589] alg: No test for adler32 (adler32-zlib) [ 3667.208884] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3667.454385] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3669.103160] Key type lgssc registered [ 3670.060063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.579963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.600510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.987123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.799130] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3687.399168] LustreError: 170340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.419094] LustreError: 170340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3689.096776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.361469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3693.417890] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3699.702924] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3703.780173] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3708.878437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.159556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.404192] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3718.110634] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3722.479625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3727.844080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3727.845746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.852948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3727.858446] Lustre: Skipped 1 previous similar message [ 3728.668981] Lustre: server umount lustre-OST0000 complete [ 3732.962757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.970087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3732.976909] Lustre: Skipped 1 previous similar message [ 3732.980299] Lustre: Skipped 1 previous similar message [ 3737.857463] Lustre: server umount lustre-MDT0000 complete [ 3738.082909] LustreError: 171066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.096710] LustreError: 171066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3740.576914] LustreError: 170320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907059 with bad export cookie 17771517372838599227 [ 3740.580222] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3740.582989] LustreError: 170320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3740.802343] Lustre: server umount lustre-MDT0001 complete [ 3749.382581] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3751.743300] Key type lgssc unregistered [ 3752.023859] LNet: 173869:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.034590] LNetError: 173869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3752.063456] LNet: Removed LNI 192.168.204.119@tcp [ 3752.781485] Key type .llcrypt unregistered [ 3752.783800] Key type ._llcrypt unregistered [ 3777.115957] Key type ._llcrypt registered [ 3777.118947] Key type .llcrypt registered [ 3777.185740] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3777.797973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3777.861084] alg: No test for adler32 (adler32-zlib) [ 3778.866259] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3779.091708] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3780.759398] Key type lgssc registered [ 3781.595570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3787.146733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3787.158453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.517966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.341953] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3797.484712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.734652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3800.713728] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3805.914303] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3809.944504] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3814.970356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3815.260733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.320264] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3826.840850] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3833.716523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3843.041229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3843.054109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.061068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.066703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3844.075878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3844.077635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3844.084106] Lustre: Skipped 1 previous similar message [ 3849.187478] LustreError: 177372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.198735] LustreError: 177372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3849.247941] Lustre: server umount lustre-OST0000 complete [ 3854.304685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.306098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3854.317584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.319192] Lustre: Skipped 1 previous similar message [ 3859.424798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3859.432041] Lustre: Skipped 1 previous similar message [ 3864.545771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3864.552676] Lustre: Skipped 1 previous similar message [ 3867.104917] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3867.227373] Lustre: server umount lustre-MDT0000 complete [ 3869.667370] LustreError: 175544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.682442] LustreError: 175544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3870.047620] LustreError: 177392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907188 with bad export cookie 7266837116446109951 [ 3870.052048] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3870.057665] LustreError: 177392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3870.341438] Lustre: server umount lustre-MDT0001 complete [ 3879.073988] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3881.149339] Key type lgssc unregistered [ 3881.389357] LNet: 179105:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3881.394345] LNetError: 179105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3882.478180] LNet: Removed LNI 192.168.204.119@tcp [ 3883.200290] Key type .llcrypt unregistered [ 3883.202805] Key type ._llcrypt unregistered [ 3900.739064] Key type ._llcrypt registered [ 3900.741781] Key type .llcrypt registered [ 3900.849191] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 3901.783379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3901.805967] alg: No test for adler32 (adler32-zlib) [ 3902.788545] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 3902.974564] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 3904.639199] Key type lgssc registered [ 3905.595070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.119584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.142756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.458165] Lustre: *** cfs_fail_loc=135, val=0*** [ 3915.547479] LustreError: 180609:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3915.553803] LustreError: 180609:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.119@tcp: cfg command failed: rc = -2 [ 3915.561176] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3915.561176] [ 3915.572688] LustreError: MGC192.168.204.119@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3915.583493] LustreError: 180570:0:(tgt_mount.c:1966:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3915.592767] LustreError: 180570:0:(tgt_mount.c:2703:server_fill_super()) Unable to start targets: -2 [ 3915.601468] LustreError: 180570:0:(tgt_mount.c:2221:server_put_super()) no obd lustre-MDT0000 [ 3915.606338] LustreError: 180570:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3915.690971] Lustre: server umount lustre-MDT0000 complete [ 3915.695400] LustreError: 180570:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3920.894551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.185801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.539465] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3930.559220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.840965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3933.745219] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 3938.879925] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3943.124308] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3946.470445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.471304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3946.478727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.483439] Lustre: Skipped 1 previous similar message [ 3951.584725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3951.587887] Lustre: Skipped 2 previous similar messages [ 3955.684550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3955.689580] Lustre: Skipped 1 previous similar message [ 3959.263133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3959.572901] Lustre: server umount lustre-MDT0000 complete [ 3960.801615] LustreError: 180980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.829500] LustreError: 180980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3965.353201] LustreError: 181714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907284 with bad export cookie 6874980092643639201 [ 3965.359526] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.377988] LustreError: 181714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3966.390914] Lustre: server umount lustre-MDT0001 complete [ 3980.676314] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 3984.865768] Key type lgssc unregistered [ 3985.315946] LNet: 183303:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3985.324323] LNetError: 183303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3985.337085] LNet: Removed LNI 192.168.204.119@tcp [ 3986.733236] Key type .llcrypt unregistered [ 3986.735677] Key type ._llcrypt unregistered [ 4017.869933] Key type ._llcrypt registered [ 4017.881773] Key type .llcrypt registered [ 4018.068086] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4018.445327] Key type .llcrypt unregistered [ 4018.448164] Key type ._llcrypt unregistered [ 4037.085596] Key type ._llcrypt registered [ 4037.087247] Key type .llcrypt registered [ 4037.290736] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4038.761561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.813209] alg: No test for adler32 (adler32-zlib) [ 4039.905937] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4040.199588] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4041.903177] Key type lgssc registered [ 4043.478519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4051.563704] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4051.609357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4067.103490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4072.224596] LustreError: 185446:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.119@tcp: failed processing log, type 4: rc = -110 [ 4098.911445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.515865] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4112.730861] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4119.675817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4119.827702] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.846939] Lustre: MGS: Client dccd5dd0-9213-45d8-b4af-8da3b3714354 (at 0@lo) reconnecting [ 4119.854405] Lustre: MGC192.168.204.119@tcp: Connection restored to 0@lo (at 0@lo) [ 4120.271540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.338352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4124.760958] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4125.666070] LustreError: 185472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.693896] LustreError: 185472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4130.787208] LustreError: 185471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.599092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.861408] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4134.088967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4139.032808] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4147.354163] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4153.488838] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4156.762631] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4170.219890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.228443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.233425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.233438] Lustre: Skipped 1 previous similar message [ 4170.266238] Lustre: Skipped 1 previous similar message [ 4174.284067] Lustre: server umount lustre-OST0000 complete [ 4180.448644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4180.449635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.465562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4180.480318] Lustre: Skipped 1 previous similar message [ 4180.489282] Lustre: Skipped 2 previous similar messages [ 4185.572613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.586072] Lustre: Skipped 1 previous similar message [ 4190.698203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4190.709209] Lustre: Skipped 1 previous similar message [ 4192.223268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4192.382668] Lustre: server umount lustre-MDT0000 complete [ 4195.808718] LustreError: 186487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.829202] LustreError: 186487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4196.104294] LustreError: 185453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907514 with bad export cookie 3808474831428938628 [ 4196.107106] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.131463] LustreError: 185453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4196.504396] Lustre: server umount lustre-MDT0001 complete [ 4207.798206] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4210.488650] Key type lgssc unregistered [ 4210.789508] LNet: 189414:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4210.804581] LNetError: 189414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4210.826680] LNet: Removed LNI 192.168.204.119@tcp [ 4211.632152] Key type .llcrypt unregistered [ 4211.633550] Key type ._llcrypt unregistered [ 4234.565287] Key type ._llcrypt registered [ 4234.566920] Key type .llcrypt registered [ 4234.661340] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4235.516439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4235.609089] alg: No test for adler32 (adler32-zlib) [ 4236.696553] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4236.898255] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4238.559176] Key type lgssc registered [ 4239.619758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4247.521277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4247.540909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.026869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.976751] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4258.276861] LustreError: 190827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.296549] LustreError: 190827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4260.997533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.359718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.649921] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4273.813898] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.706413] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4286.729571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.892501] LustreError: 192647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.112125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.997996] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4293.633574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4300.648996] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4306.058932] Lustre: Failing over lustre-MDT0000 [ 4306.253331] Lustre: server umount lustre-MDT0000 complete [ 4307.428448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4307.434816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.465166] LustreError: 190828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.971155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.932868] LustreError: 190823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.949591] LustreError: 190823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4319.045252] LustreError: 190824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.069702] LustreError: 190824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4325.160039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.335244] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.344961] Lustre: 190138:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788907627/real 1788907627] req@00000000ac9efb24 x1875805528418432/t0(0) o400->MGC192.168.204.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788907643 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4325.704681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.770645] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4329.222127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4330.712487] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4330.998723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4331.038903] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4331.086801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4335.569562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4341.024555] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4348.897589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4348.905499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.913742] Lustre: Skipped 1 previous similar message [ 4348.918542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.295265] Lustre: server umount lustre-OST0000 complete [ 4356.577516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.585281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4356.589778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4361.700046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4361.705069] Lustre: Skipped 2 previous similar messages [ 4362.837493] Lustre: server umount lustre-MDT0000 complete [ 4365.637458] LustreError: 194406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907684 with bad export cookie 8787964897429508416 [ 4365.640202] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.645217] LustreError: 194406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4365.903939] Lustre: server umount lustre-MDT0001 complete [ 4373.785611] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4375.848398] Key type lgssc unregistered [ 4376.080469] LNet: 195664:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.090375] LNetError: 195664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4376.108598] LNet: Removed LNI 192.168.204.119@tcp [ 4376.734631] Key type .llcrypt unregistered [ 4376.736274] Key type ._llcrypt unregistered [ 4394.084307] Key type ._llcrypt registered [ 4394.086141] Key type .llcrypt registered [ 4394.166784] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4394.766919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4394.776058] alg: No test for adler32 (adler32-zlib) [ 4395.696623] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4395.862669] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4397.487169] Key type lgssc registered [ 4398.260753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4403.459243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4403.479743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.746554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.600477] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4413.405388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.650936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4416.578141] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4421.735297] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4425.633162] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4429.769659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4429.991746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.032387] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4435.441230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4438.635059] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4441.926295] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4445.928323] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4460.097191] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4466.144850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4466.150857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.161218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4469.216521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4469.220635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.221570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4469.238346] Lustre: Skipped 1 previous similar message [ 4471.615490] Lustre: server umount lustre-OST0000 complete [ 4474.338647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.347276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4474.350801] Lustre: Skipped 1 previous similar message [ 4479.457100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4479.460358] Lustre: Skipped 2 previous similar messages [ 4484.580140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.590347] Lustre: Skipped 1 previous similar message [ 4488.159354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4488.270465] Lustre: server umount lustre-MDT0000 complete [ 4489.697057] LustreError: 197050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.714319] LustreError: 197050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4491.124652] LustreError: 199587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907810 with bad export cookie 3437412321169891876 [ 4491.127046] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4491.137954] LustreError: 199587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4491.364862] Lustre: server umount lustre-MDT0001 complete [ 4499.299827] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4500.964868] Key type lgssc unregistered [ 4501.142071] LNet: 200925:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4501.146394] LNetError: 200925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4501.159412] LNet: Removed LNI 192.168.204.119@tcp [ 4501.684534] Key type .llcrypt unregistered [ 4501.686218] Key type ._llcrypt unregistered [ 4514.741296] Key type ._llcrypt registered [ 4514.745982] Key type .llcrypt registered [ 4514.815893] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4515.361749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4515.383065] alg: No test for adler32 (adler32-zlib) [ 4516.339307] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4516.541099] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4518.167166] Key type lgssc registered [ 4519.048514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4524.973225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4524.985482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.280392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.540524] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4535.739350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.991542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4539.536166] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4544.854179] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.733419] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4553.619986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4553.874205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.416652] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4563.584104] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4571.317698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4578.650309] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4586.557839] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4593.480886] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4602.952354] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4612.583037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4612.588694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.591445] LustreError: Skipped 1 previous similar message [ 4612.595252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4612.606072] Lustre: Skipped 1 previous similar message [ 4612.623282] Lustre: Skipped 1 previous similar message [ 4617.699525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4617.704023] Lustre: Skipped 1 previous similar message [ 4622.822632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.832277] Lustre: Skipped 1 previous similar message [ 4624.354968] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4624.444702] Lustre: server umount lustre-OST0000 complete [ 4627.937303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.938968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.947152] Lustre: Skipped 1 previous similar message [ 4632.863946] Lustre: server umount lustre-MDT0000 complete [ 4633.058706] LustreError: 202163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.069105] LustreError: 202163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4635.300417] LustreError: 202148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788907954 with bad export cookie 12177976723516582325 [ 4635.304621] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.308496] LustreError: 202148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4635.540940] Lustre: server umount lustre-MDT0001 complete [ 4641.650141] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4643.192809] Key type lgssc unregistered [ 4643.383423] LNet: 207160:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4643.390686] LNetError: 207160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4643.401490] LNet: Removed LNI 192.168.204.119@tcp [ 4643.907517] Key type .llcrypt unregistered [ 4643.909535] Key type ._llcrypt unregistered [ 4658.626611] Key type ._llcrypt registered [ 4658.628510] Key type .llcrypt registered [ 4658.717406] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4659.230137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4659.312644] alg: No test for adler32 (adler32-zlib) [ 4660.209446] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4660.362449] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4661.983178] Key type lgssc registered [ 4662.661304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4667.517852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4667.533983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.859186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.917974] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4677.448477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.812065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4680.662123] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4685.591488] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4689.138556] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4693.288749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.518469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.572638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4697.502833] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4701.888249] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4712.934307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.938551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4712.944996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4712.959580] Lustre: Skipped 1 previous similar message [ 4717.399981] Lustre: server umount lustre-OST0000 complete [ 4723.171801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.180484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4723.181065] Lustre: Skipped 2 previous similar messages [ 4723.187862] Lustre: Skipped 1 previous similar message [ 4726.369900] Lustre: server umount lustre-MDT0000 complete [ 4728.290767] LustreError: 208550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.306431] LustreError: 208550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4728.711825] LustreError: 211277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908047 with bad export cookie 8842671925498048236 [ 4728.715609] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4728.720265] LustreError: 211277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4728.930467] Lustre: server umount lustre-MDT0001 complete [ 4735.705887] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4737.336331] Key type lgssc unregistered [ 4737.513565] LNet: 212229:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.520164] LNetError: 212229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.533720] LNet: Removed LNI 192.168.204.119@tcp [ 4738.047799] Key type .llcrypt unregistered [ 4738.049836] Key type ._llcrypt unregistered [ 4750.215812] Key type ._llcrypt registered [ 4750.217534] Key type .llcrypt registered [ 4750.284249] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4750.892616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4750.905916] alg: No test for adler32 (adler32-zlib) [ 4751.863647] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4752.012102] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4753.647178] Key type lgssc registered [ 4754.275082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4758.466599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4758.479586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.735348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.088602] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4767.847319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.139779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4771.096371] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4776.337699] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4780.409230] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4784.681310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4784.906842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.812806] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4794.163703] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4804.396149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4814.817100] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.821699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4814.824463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.830314] Lustre: Skipped 1 previous similar message [ 4819.938387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4819.946198] Lustre: Skipped 2 previous similar messages [ 4820.290367] Lustre: server umount lustre-OST0000 complete [ 4824.545206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4824.550140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4824.559937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.181348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.185963] Lustre: Skipped 3 previous similar messages [ 4835.296276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4835.301738] Lustre: Skipped 1 previous similar message [ 4837.346075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4837.447370] Lustre: server umount lustre-MDT0000 complete [ 4839.871190] LustreError: 213453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908158 with bad export cookie 7555019106949028781 [ 4839.874123] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4839.879551] LustreError: 213453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4840.084746] Lustre: server umount lustre-MDT0001 complete [ 4846.186861] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 4847.911530] Key type lgssc unregistered [ 4848.109207] LNet: 217180:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4848.113850] LNetError: 217180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4848.126612] LNet: Removed LNI 192.168.204.119@tcp [ 4848.668912] Key type .llcrypt unregistered [ 4848.675379] Key type ._llcrypt unregistered [ 4861.568164] Key type ._llcrypt registered [ 4861.570201] Key type .llcrypt registered [ 4861.657971] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 4862.422912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4862.439234] alg: No test for adler32 (adler32-zlib) [ 4863.424547] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 4863.589871] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 4865.239264] Key type lgssc registered [ 4865.929385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4870.808157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4870.834883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.132407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.573124] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4879.972694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.251139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.165287] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4888.551787] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4892.424240] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4896.689748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.931147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.314533] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4904.880402] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4916.391381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4921.463336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4921.522941] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4921.671299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4925.512990] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 4931.278902] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4932.482231] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4943.634288] Lustre: Permanently deactivating lustre-OST0001 [ 4943.649649] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4952.101371] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4957.687905] Lustre: Permanently reactivating lustre-OST0001 [ 4957.709339] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4957.716540] Lustre: Skipped 2 previous similar messages [ 4959.319377] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.330825] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4959.337396] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4959.345324] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4959.368105] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4969.952089] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4969.958969] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.963371] LustreError: Skipped 1 previous similar message [ 4969.965914] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4969.976058] Lustre: Skipped 2 previous similar messages [ 4969.991293] Lustre: Skipped 1 previous similar message [ 4972.519280] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4972.525026] Lustre: Skipped 1 previous similar message [ 4974.735576] Lustre: server umount lustre-OST0001 complete [ 4977.641441] LustreError: 221367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4977.644484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.657149] LustreError: 221367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4977.669357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4982.241942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4982.248744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.257317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4982.759530] LustreError: 220244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4982.766720] LustreError: 220244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4983.308193] Lustre: server umount lustre-OST0000 complete [ 4987.875756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.875789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4987.878987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4987.878994] Lustre: Skipped 1 previous similar message [ 4987.884851] Lustre: Skipped 1 previous similar message [ 4998.116155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4998.120594] Lustre: Skipped 4 previous similar messages [ 5000.159131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5000.282294] Lustre: server umount lustre-MDT0000 complete [ 5002.211637] LustreError: 218426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.218693] LustreError: 218426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5002.662746] LustreError: 220270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908321 with bad export cookie 8471205015372679218 [ 5002.664749] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5002.671777] LustreError: 220270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5002.903340] Lustre: server umount lustre-MDT0001 complete [ 5009.850848] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5011.523844] Key type lgssc unregistered [ 5011.689501] LNet: 223489:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5011.699325] LNetError: 223489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5011.713083] LNet: Removed LNI 192.168.204.119@tcp [ 5012.311157] Key type .llcrypt unregistered [ 5012.312660] Key type ._llcrypt unregistered [ 5036.540785] Key type ._llcrypt registered [ 5036.543759] Key type .llcrypt registered [ 5036.610893] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5037.223662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5037.252174] alg: No test for adler32 (adler32-zlib) [ 5038.229891] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5038.380173] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 5040.009307] Key type lgssc registered [ 5040.765503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5045.566251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5045.578155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.763871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5046.783723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5046.914127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5049.780611] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5055.085988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.149973] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5055.167829] Lustre: 225671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5055.363183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.510876] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5061.838834] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5065.186327] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5069.126309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5069.250777] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5069.390624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5071.068665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 5073.147155] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5078.102853] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5080.186763] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5087.134719] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5095.165349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5101.089230] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5113.816329] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5113.821104] Lustre: Skipped 1 previous similar message [ 5135.041641] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5135.048322] Lustre: Skipped 2 previous similar messages [ 5174.255221] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5174.262872] Lustre: Skipped 4 previous similar messages [ 5239.492084] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 5239.497631] Lustre: Skipped 7 previous similar messages [ 5245.412246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.412635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.420049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5245.420975] Lustre: Skipped 1 previous similar message [ 5245.427482] Lustre: Skipped 1 previous similar message [ 5250.533550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5250.538069] Lustre: Skipped 1 previous similar message [ 5255.660984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5258.207170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5258.473877] Lustre: server umount lustre-OST0000 complete [ 5265.032084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.081040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5265.100087] Lustre: Skipped 2 previous similar messages [ 5270.503359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.515035] Lustre: Skipped 1 previous similar message [ 5271.786966] Lustre: server umount lustre-MDT0000 complete [ 5278.102983] LustreError: 228236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908596 with bad export cookie 5667228104452556822 [ 5278.107197] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5278.132592] LustreError: 228236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5278.963699] Lustre: server umount lustre-MDT0001 complete [ 5298.254283] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5301.956765] Key type lgssc unregistered [ 5302.216617] LNet: 230493:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5302.224406] LNetError: 230493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5302.241462] LNet: Removed LNI 192.168.204.119@tcp [ 5303.491919] Key type .llcrypt unregistered [ 5303.502447] Key type ._llcrypt unregistered [ 5332.294416] Key type ._llcrypt registered [ 5332.296437] Key type .llcrypt registered [ 5332.390251] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5333.316483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5333.446588] alg: No test for adler32 (adler32-zlib) [ 5334.599763] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5334.884880] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 5336.599683] Key type lgssc registered [ 5337.699973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5346.259134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5346.309831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.747874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.298768] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5358.050578] LustreError: 231908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.062724] LustreError: 231908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5360.819163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.145349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5365.639495] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5373.067850] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5378.285623] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5384.592922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5384.916898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.406721] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_default_debug -1 all [ 5393.166354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 5399.000390] Lustre: DEBUG MARKER: oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5416.940717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5416.949435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.979378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5417.286538] Lustre: server umount lustre-OST0000 complete [ 5422.561170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5422.572843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5422.598416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.097496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.101892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5424.124819] Lustre: Skipped 1 previous similar message [ 5429.216600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5429.227406] Lustre: Skipped 1 previous similar message [ 5433.315680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5433.321830] Lustre: Skipped 1 previous similar message [ 5435.360230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5435.595408] Lustre: server umount lustre-MDT0000 complete [ 5438.435492] LustreError: 231903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.447029] LustreError: 231903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5439.714341] LustreError: 231889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788908758 with bad export cookie 8948915562621484335 [ 5439.722664] LustreError: MGC192.168.204.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5439.729868] LustreError: 231889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5440.070529] Lustre: server umount lustre-MDT0001 complete [ 5452.572813] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5455.589339] Key type lgssc unregistered [ 5455.911642] LNet: 235598:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5455.922615] LNetError: 235598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5457.018565] LNet: Removed LNI 192.168.204.119@tcp [ 5457.957133] Key type .llcrypt unregistered [ 5457.959746] Key type ._llcrypt unregistered [ 5480.710469] Key type ._llcrypt registered [ 5480.714985] Key type .llcrypt registered [ 5480.863173] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5481.828254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5481.867877] alg: No test for adler32 (adler32-zlib) [ 5482.891989] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5483.190490] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 5484.879214] Key type lgssc registered [ 5486.142680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5757.184961] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5759.631282] Key type lgssc unregistered [ 5759.851088] LNet: 237623:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5759.868357] LNetError: 237623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5759.888921] LNet: Removed LNI 192.168.204.119@tcp [ 5760.627710] Key type .llcrypt unregistered [ 5760.630148] Key type ._llcrypt unregistered [ 5786.072629] Key type ._llcrypt registered [ 5786.074493] Key type .llcrypt registered [ 5786.185763] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing set_hostid [ 5801.492557] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing load_modules_local [ 5802.408715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5802.505626] alg: No test for adler32 (adler32-zlib) [ 5803.624693] Lustre: Lustre: Build Version: 2.17.55_26_gfb964ff [ 5803.914840] LNet: Added LNI 192.168.204.119@tcp [8/256/0/180] [ 5805.583601] Key type lgssc registered [ 5806.875174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5836.061709] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5626 sec ========== 19:12:33 (1788909153) [ 5838.037443] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:12:34 (1788909154) === [ 5841.748074] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:12:38 (1788909158) === [ 5875.718944] Lustre: DEBUG MARKER: oleg419-server.virtnet: executing unload_modules_local [ 5877.646715] Key type lgssc unregistered [ 5877.825737] LNet: 242861:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5877.837376] LNetError: 242861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5877.851687] LNet: Removed LNI 192.168.204.119@tcp [ 5878.385235] Key type .llcrypt unregistered [ 5878.388412] Key type ._llcrypt unregistered