[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481207659 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009024] pid_max: default: 32768 minimum: 301 [ 0.010149] LSM: Security Framework initializing [ 0.011084] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.014018] *** VALIDATE selinux *** [ 0.022392] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026746] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.031356] *** VALIDATE proc *** [ 0.032247] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037020] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042913] debug: unmapping init [mem 0xffffffff85459000-0xffffffff85460fff] [ 0.044174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046028] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051030] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055520] smp: Bringing up secondary CPUs ... [ 0.056625] x86: Booting SMP configuration: [ 0.057032] .... node #0, CPUs: #1 #2 #3 [ 0.060725] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147259] node 0 deferred pages initialised in 80ms [ 0.151322] devtmpfs: initialized [ 0.152347] x86/mm: Memory block size: 128MB [ 0.155593] gcov: version magic: 0x41383552 [ 0.159327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164377] pinctrl core: initialized pinctrl subsystem [ 0.167214] [ 0.167847] ************************************************************* [ 0.170014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172011] ** ** [ 0.174013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.177014] ** ** [ 0.179012] ** This means that this kernel is built to expose internal ** [ 0.182037] ** IOMMU data structures, which may compromise security on ** [ 0.184013] ** your system. ** [ 0.186011] ** ** [ 0.188015] ** If you see this message and you are not debugging the ** [ 0.191013] ** kernel, report this immediately to your vendor! ** [ 0.193013] ** ** [ 0.195012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196000] ************************************************************* [ 0.198743] NET: Registered protocol family 16 [ 0.200502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209156] cpuidle: using governor menu [ 0.211878] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214823] PCI: Using configuration type 1 for base access [ 0.217152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228139] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229820] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231141] cryptd: max_cpu_qlen set to 1000 [ 0.233162] ACPI: Added _OSI(Module Device) [ 0.234029] ACPI: Added _OSI(Processor Device) [ 0.235013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237033] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242831] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248464] ACPI: Interpreter enabled [ 0.249000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252018] ACPI: Using IOAPIC for interrupt routing [ 0.253109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256512] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265109] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280366] acpiphp: Slot [2] registered [ 0.282093] acpiphp: Slot [5] registered [ 0.284696] acpiphp: Slot [6] registered [ 0.286189] acpiphp: Slot [7] registered [ 0.287174] acpiphp: Slot [8] registered [ 0.288137] acpiphp: Slot [9] registered [ 0.290118] acpiphp: Slot [10] registered [ 0.291271] acpiphp: Slot [3] registered [ 0.293106] acpiphp: Slot [4] registered [ 0.294170] acpiphp: Slot [11] registered [ 0.296115] acpiphp: Slot [12] registered [ 0.297165] acpiphp: Slot [13] registered [ 0.299137] acpiphp: Slot [14] registered [ 0.300152] acpiphp: Slot [15] registered [ 0.302177] acpiphp: Slot [16] registered [ 0.303120] acpiphp: Slot [17] registered [ 0.305175] acpiphp: Slot [18] registered [ 0.306215] acpiphp: Slot [19] registered [ 0.308125] acpiphp: Slot [20] registered [ 0.310118] acpiphp: Slot [21] registered [ 0.311120] acpiphp: Slot [22] registered [ 0.312000] acpiphp: Slot [23] registered [ 0.314130] acpiphp: Slot [24] registered [ 0.318142] acpiphp: Slot [25] registered [ 0.320123] acpiphp: Slot [26] registered [ 0.321140] acpiphp: Slot [27] registered [ 0.323101] acpiphp: Slot [28] registered [ 0.324090] acpiphp: Slot [29] registered [ 0.326089] acpiphp: Slot [30] registered [ 0.327092] acpiphp: Slot [31] registered [ 0.328067] PCI host bridge to bus 0000:00 [ 0.330024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335051] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341275] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346240] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350219] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353486] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364840] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369503] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381188] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383953] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389967] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.396018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422335] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.457017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465283] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.482019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.497016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.511595] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580513] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.588017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.594019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.621013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.633922] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.647017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.652017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.669018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.682833] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.691018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.699015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.717019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728277] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.730435] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.733392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.736421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.738357] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.743215] iommu: Default domain type: Passthrough [ 0.745816] SCSI subsystem initialized [ 0.747249] ACPI: bus type USB registered [ 0.749175] usbcore: registered new interface driver usbfs [ 0.752144] usbcore: registered new interface driver hub [ 0.754193] usbcore: registered new device driver usb [ 0.757254] pps_core: LinuxPPS API ver. 1 registered [ 0.759015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.763101] PTP clock support registered [ 0.766053] EDAC MC: Ver: 3.0.0 [ 0.768209] PCI: Using ACPI for IRQ routing [ 0.769630] NetLabel: Initializing [ 0.771026] NetLabel: domain hash size = 128 [ 0.773114] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.775125] NetLabel: unlabeled traffic allowed by default [ 0.778179] vgaarb: loaded [ 0.780345] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.782026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.787382] clocksource: Switched to clocksource kvm-clock [ 0.890575] VFS: Disk quotas dquot_6.6.0 [ 0.892142] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.894054] *** VALIDATE ramfs *** [ 0.895023] *** VALIDATE hugetlbfs *** [ 0.896217] pnp: PnP ACPI init [ 0.899215] pnp: PnP ACPI: found 6 devices [ 0.916158] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.919621] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.921994] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.924348] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.926775] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.929136] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.932031] NET: Registered protocol family 2 [ 0.934489] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.939177] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.942344] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947715] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950988] TCP: Hash tables configured (established 65536 bind 65536) [ 0.954093] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.956961] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960811] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963737] NET: Registered protocol family 1 [ 0.966355] RPC: Registered named UNIX socket transport module. [ 0.968281] RPC: Registered udp transport module. [ 0.970015] RPC: Registered tcp transport module. [ 0.971761] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.974176] NET: Registered protocol family 44 [ 0.975798] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.977404] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.979316] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981716] PCI: CLS 0 bytes, default 64 [ 0.984219] Unpacking initramfs... [ 2.383765] debug: unmapping init [mem 0xffff93edfcc54000-0xffff93edfffbffff] [ 2.387699] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.390769] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.394297] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.899837] Initialise system trusted keyrings [ 2.901443] Key type blacklist registered [ 2.903345] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.914309] zbud: loaded [ 2.917234] *** VALIDATE nfs *** [ 2.918651] *** VALIDATE nfs4 *** [ 2.920697] pstore: using deflate compression [ 2.927991] Platform Keyring initialized [ 3.029664] NET: Registered protocol family 38 [ 3.031672] Key type asymmetric registered [ 3.033157] Asymmetric key parser 'x509' registered [ 3.035197] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.038571] io scheduler mq-deadline registered [ 3.040442] io scheduler kyber registered [ 3.042178] io scheduler bfq registered [ 3.044165] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.046922] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.049706] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.052480] ACPI: Power Button [PWRF] [ 3.057855] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.064868] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.079617] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.088085] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.103533] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.130753] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.159229] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.165584] Non-volatile memory driver v1.3 [ 3.167367] Linux agpgart interface v0.103 [ 3.197781] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.200581] vda: detected capacity change from 0 to 69001216 [ 3.221935] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.225057] vdb: detected capacity change from 0 to 1073741824 [ 3.241766] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.244278] vdc: detected capacity change from 0 to 2621440000 [ 3.261062] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.264477] vdd: detected capacity change from 0 to 2621440000 [ 3.282107] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.285762] vde: detected capacity change from 0 to 4294967296 [ 3.319974] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.323237] vdf: detected capacity change from 0 to 4294967296 [ 3.338978] libphy: Fixed MDIO Bus: probed [ 3.344863] usbcore: registered new interface driver usbserial_generic [ 3.347757] usbserial: USB Serial support registered for generic [ 3.349923] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.354422] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.356275] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.358478] mousedev: PS/2 mouse device common for all mice [ 3.361530] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.363785] rtc_cmos 00:05: RTC can wake from S4 [ 3.367658] rtc_cmos 00:05: registered as rtc0 [ 3.369451] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.370819] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.371886] intel_pstate: CPU model not supported [ 3.376789] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.376873] hid: raw HID events driver (C) Jiri Kosina [ 3.380682] usbcore: registered new interface driver usbhid [ 3.382513] usbhid: USB HID core driver [ 3.384182] drop_monitor: Initializing network drop monitor service [ 3.386152] Initializing XFRM netlink socket [ 3.388158] NET: Registered protocol family 10 [ 3.391572] Segment Routing with IPv6 [ 3.392952] NET: Registered protocol family 17 [ 3.396246] mpls_gso: MPLS GSO support [ 3.403303] RAS: Correctable Errors collector initialized. [ 3.405556] AVX version of gcm_enc/dec engaged. [ 3.407331] AES CTR mode by8 optimization enabled [ 3.490089] sched_clock: Marking stable (3490040660, 0)->(4394229507, -904188847) [ 3.493938] registered taskstats version 1 [ 3.496405] Loading compiled-in X.509 certificates [ 3.498671] zswap: loaded using pool lzo/zbud [ 3.525698] Key type big_key registered [ 3.537480] Key type encrypted registered [ 3.538919] ima: No TPM chip found, activating TPM-bypass! [ 3.540668] ima: Allocated hash algorithm: sha1 [ 3.541920] ima: No architecture policies found [ 3.542975] evm: Initialising EVM extended attributes: [ 3.544119] evm: security.selinux [ 3.545038] evm: security.ima [ 3.546074] evm: security.capability [ 3.547304] evm: HMAC attrs: 0x1 [ 3.550189] rtc_cmos 00:05: setting system clock to 2026-05-25 10:47:00 UTC (1779706020) [ 3.556798] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 3.560027] debug: unmapping init [mem 0xffffffff85182000-0xffffffff85458fff] [ 3.564315] Write protecting the kernel read-only data: 28672k [ 3.567428] debug: unmapping init [mem 0xffffffff83803000-0xffffffff839fffff] [ 3.570249] debug: unmapping init [mem 0xffffffff84114000-0xffffffff841fffff] [ 3.603774] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.611749] systemd[1]: Detected virtualization kvm. [ 3.613717] systemd[1]: Detected architecture x86-64. [ 3.615521] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.640642] systemd[1]: No hostname configured. [ 3.642677] systemd[1]: Set hostname to . [ 3.645060] random: systemd: uninitialized urandom read (16 bytes read) [ 3.647748] systemd[1]: Initializing machine ID from random generator. [ 3.709239] random: ln: uninitialized urandom read (6 bytes read) [ 3.800047] random: systemd: uninitialized urandom read (16 bytes read) [ 3.803092] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.808362] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.812839] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.524368] device-mapper: uevent: version 1.0.3 [ 4.526877] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.262688] random: fast init done [ 5.265431] virtio_net virtio0 ens2: renamed from eth0 [ 5.341119] scsi host0: ata_piix [ 5.363980] scsi host1: ata_piix [ 5.365674] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.368419] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.738378] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.052418] random: crng init done [ 10.053670] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.529901] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.718331] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.985097] SELinux: Disabled at runtime. [ 12.047400] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.055849] systemd[1]: Detected virtualization kvm. [ 12.057935] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.588474] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.592122] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.597240] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.603344] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.607651] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.615275] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.619460] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Jour[ 12.786156] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.059526] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.375271] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.473835] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.560857] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.572234] EDAC sbridge: Ver: 1.1.2 [ 15.280717] Key type dns_resolver registered [ 15.584860] NFS: Registering the id_resolver key type [ 15.587187] Key type id_resolver registered [ 15.589196] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ 23.298423] hrtimer: interrupt took 18788548 ns Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg448-server login: [ 69.971577] libcfs: loading out-of-tree module taints kernel. [ 70.010395] Key type ._llcrypt registered [ 70.014476] Key type .llcrypt registered [ 70.140556] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 89.767672] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 91.430088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.450875] alg: No test for adler32 (adler32-zlib) [ 93.072129] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 94.283751] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 96.175414] Key type lgssc registered [ 98.258640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.835100] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.404909] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 178.887850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.904723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.195614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.238413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.368955] Lustre: lustre-MDT0000: new disk, initializing [ 180.496107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.521956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.795587] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.512879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.645892] Lustre: 6522:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.683834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.688935] Lustre: Skipped 1 previous similar message [ 199.804514] Lustre: lustre-MDT0001: new disk, initializing [ 199.882135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.919755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.937232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.994484] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.937867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.334044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.680844] Lustre: lustre-OST0000: new disk, initializing [ 220.686091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.696148] Lustre: 8427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.772348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.380278] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.934151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.943614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.073053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.319652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.582986] Lustre: lustre-OST0001: new disk, initializing [ 244.598393] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.612131] Lustre: 9484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.734127] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.900338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.909290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.999299] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.673493] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.479854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.531478] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 281.400764] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.600574] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing yml_node [ 291.950808] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 294.827604] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 297.388177] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 299.159508] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 06:51:55 EDT 2026 [ 323.000959] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 325.361082] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 343.017528] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.021345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 343.033045] Lustre: Skipped 1 previous similar message [ 348.134963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 348.142268] Lustre: Skipped 5 previous similar messages [ 348.581565] Lustre: server umount lustre-MDT0000 complete [ 352.641632] LustreError: 7453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706369 with bad export cookie 14158251209164638107 [ 352.645241] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 352.680564] LustreError: 7453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 353.255712] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.262889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.281332] Lustre: Skipped 3 previous similar messages [ 353.293890] Lustre: Skipped 1 previous similar message [ 357.295760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.125508] Lustre: server umount lustre-MDT0001 complete [ 364.209241] Lustre: server umount lustre-OST0000 complete [ 368.651266] Lustre: server umount lustre-OST0001 complete [ 376.510970] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 387.548687] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 433.364229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.855773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.928742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.108835] Lustre: lustre-MDT0000: new disk, initializing [ 434.287275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.432637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.996651] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 453.593460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.823983] Lustre: 15000:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 453.961611] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 453.967039] Lustre: Skipped 1 previous similar message [ 454.093812] Lustre: lustre-MDT0001: new disk, initializing [ 454.292397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.361147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.373123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 460.299608] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 470.390499] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 475.163404] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.446741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.919418] Lustre: lustre-OST0000: new disk, initializing [ 483.926161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 483.937228] Lustre: 16744:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 484.088715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.006084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 485.025187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 485.131646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.027878] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 499.864831] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 504.972670] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 505.252412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 510.416683] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 510.649801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 516.067734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.073724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.075417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.075424] Lustre: Skipped 1 previous similar message [ 516.122695] Lustre: Skipped 1 previous similar message [ 526.306519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.316340] Lustre: Skipped 4 previous similar messages [ 527.327141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.577383] Lustre: server umount lustre-OST0000 complete [ 536.032841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.044554] LustreError: Skipped 1 previous similar message [ 536.048960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.430282] Lustre: server umount lustre-MDT0000 complete [ 541.665228] LustreError: 15014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.678809] LustreError: 15014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 542.286417] LustreError: 14992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706559 with bad export cookie 14158251209164639640 [ 542.288084] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.304794] LustreError: 14992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 542.586920] Lustre: server umount lustre-MDT0001 complete [ 546.034074] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:56:02 (1779706562) [ 552.847966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.354413] LustreError: 18863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.412628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 558.026886] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 558.563478] LustreError: 18864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.689566] LustreError: 18863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.139677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.263893] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 580.414042] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 584.755210] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 592.288561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.081272] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 606.632738] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 619.491883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.501114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.511347] Lustre: Skipped 1 previous similar message [ 619.517186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.527639] LustreError: Skipped 1 previous similar message [ 619.543716] Lustre: Skipped 3 previous similar messages [ 620.932457] Lustre: server umount lustre-OST0000 complete [ 629.730924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.757647] Lustre: Skipped 1 previous similar message [ 631.712978] Lustre: server umount lustre-MDT0000 complete [ 634.849564] LustreError: 18864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.870965] LustreError: 18864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 635.742906] LustreError: 18844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706652 with bad export cookie 14158251209164640522 [ 635.751747] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.756747] LustreError: 18844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 636.212919] Lustre: server umount lustre-MDT0001 complete [ 649.322307] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 652.928343] Key type lgssc unregistered [ 653.448830] LNet: 22042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.462705] LNetError: 22042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.476929] LNet: Removed LNI 192.168.204.148@tcp [ 654.598631] Key type .llcrypt unregistered [ 654.602197] Key type ._llcrypt unregistered [ 680.417484] Key type ._llcrypt registered [ 680.432091] Key type .llcrypt registered [ 680.579627] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 681.820505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 681.894101] alg: No test for adler32 (adler32-zlib) [ 683.131399] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 683.372602] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 685.075702] Key type lgssc registered [ 686.506991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.327560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.362837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.069923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 703.169378] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 707.555696] LustreError: 23440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.628022] LustreError: 23440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 712.678079] LustreError: 23441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.455984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.865269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 718.431136] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 725.153176] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.412238] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 738.077438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 738.609349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 742.729977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 745.871790] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 753.560882] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 773.613569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 773.630871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.632476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 773.665286] Lustre: Skipped 1 previous similar message [ 778.723637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 778.753664] Lustre: Skipped 2 previous similar messages [ 783.842133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 783.858076] Lustre: Skipped 1 previous similar message [ 785.375276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 785.672563] Lustre: server umount lustre-OST0000 complete [ 790.496353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 790.516101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.548417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.090226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.655084] Lustre: server umount lustre-MDT0000 complete [ 799.210575] LustreError: 23435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.233905] LustreError: 23435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 800.840115] LustreError: 23421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706817 with bad export cookie 17930445511699720764 [ 800.844054] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.857496] LustreError: 23421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 801.315462] Lustre: server umount lustre-MDT0001 complete [ 812.932710] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 815.933748] Key type lgssc unregistered [ 816.273426] LNet: 27063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.285602] LNetError: 27063:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.319732] LNet: Removed LNI 192.168.204.148@tcp [ 818.130560] Key type .llcrypt unregistered [ 818.133117] Key type ._llcrypt unregistered [ 848.473615] Key type ._llcrypt registered [ 848.478947] Key type .llcrypt registered [ 848.743706] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 850.267160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.310797] alg: No test for adler32 (adler32-zlib) [ 851.774641] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 852.151172] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 853.879231] Key type lgssc registered [ 855.760685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.098842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.116224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.950956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.106227] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 878.131632] LustreError: 28473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.154532] LustreError: 28473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 883.180216] LustreError: 28472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.483487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.979795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 890.469233] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 898.739669] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.123232] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 920.567858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.145584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.534418] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 935.170947] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 938.564876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 948.707411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.725280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 948.737508] Lustre: Skipped 1 previous similar message [ 953.683603] Lustre: server umount lustre-OST0000 complete [ 958.945861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.949900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.961538] Lustre: Skipped 2 previous similar messages [ 958.978660] Lustre: Skipped 1 previous similar message [ 964.064797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.080339] Lustre: Skipped 1 previous similar message [ 964.760649] Lustre: server umount lustre-MDT0000 complete [ 968.182287] LustreError: 30639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.209688] LustreError: 30639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 968.996518] LustreError: 28454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779706985 with bad export cookie 16574303000585885397 [ 968.997618] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 969.022834] LustreError: 28454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 969.489599] Lustre: server umount lustre-MDT0001 complete [ 983.035137] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 986.037628] Key type lgssc unregistered [ 986.310751] LNet: 32002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.323505] LNetError: 32002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.367518] LNet: Removed LNI 192.168.204.148@tcp [ 987.293732] Key type .llcrypt unregistered [ 987.300411] Key type ._llcrypt unregistered [ 1013.852715] Key type ._llcrypt registered [ 1013.856680] Key type .llcrypt registered [ 1013.987374] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1015.058592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1015.150229] alg: No test for adler32 (adler32-zlib) [ 1016.356993] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1016.696691] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1018.384269] Key type lgssc registered [ 1019.385017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1028.064771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1028.094139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.669471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1034.924211] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1039.850457] LustreError: 33398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.886297] LustreError: 33398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1044.966179] LustreError: 33399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.181569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.659761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1053.678674] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1062.385400] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1067.286085] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1075.979119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.400090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.167450] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1089.819665] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1099.956171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1108.448786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.462500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.477400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.498101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1110.500984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.512858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.630377] Lustre: server umount lustre-OST0000 complete [ 1119.200486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1119.213662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.224982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.237593] Lustre: Skipped 1 previous similar message [ 1124.039067] Lustre: server umount lustre-MDT0000 complete [ 1125.857276] LustreError: 34109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.889967] LustreError: 34109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1127.949925] LustreError: 33377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707144 with bad export cookie 7634592175116803176 [ 1127.959158] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.963431] LustreError: 33377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1128.266826] Lustre: server umount lustre-MDT0001 complete [ 1139.261031] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1141.894619] Key type lgssc unregistered [ 1142.221675] LNet: 36598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.233280] LNetError: 36598:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.256734] LNet: Removed LNI 192.168.204.148@tcp [ 1143.042324] Key type .llcrypt unregistered [ 1143.048662] Key type ._llcrypt unregistered [ 1171.104965] Key type ._llcrypt registered [ 1171.112435] Key type .llcrypt registered [ 1171.293314] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1172.492676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.504902] alg: No test for adler32 (adler32-zlib) [ 1173.658548] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1173.932263] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1175.715878] Key type lgssc registered [ 1176.936238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1185.226932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.261153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.874586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.479727] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1197.030549] LustreError: 37996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.048676] LustreError: 37996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1201.235841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.625610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1205.952729] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1212.337292] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1216.763679] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1224.160401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1224.586653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.238592] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1231.746919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1237.521372] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1245.487168] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1247.206556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.207058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.212911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1247.215555] LustreError: Skipped 1 previous similar message [ 1247.259325] Lustre: Skipped 1 previous similar message [ 1250.583981] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1250.589653] Lustre: Skipped 1 previous similar message [ 1250.882773] Lustre: server umount lustre-OST0000 complete [ 1272.800632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.806690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.810128] Lustre: Skipped 1 previous similar message [ 1277.926052] LustreError: 39957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.945555] LustreError: 39957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1278.043683] Lustre: server umount lustre-MDT0000 complete [ 1281.980476] LustreError: 37975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707298 with bad export cookie 15326640099388335608 [ 1281.992164] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1281.998050] LustreError: 37975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1296.351681] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1296.633852] Lustre: server umount lustre-MDT0001 complete [ 1309.480663] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1312.446902] Key type lgssc unregistered [ 1312.767703] LNet: 41347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.778234] LNetError: 41347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.812684] LNet: Removed LNI 192.168.204.148@tcp [ 1313.777761] Key type .llcrypt unregistered [ 1313.782242] Key type ._llcrypt unregistered [ 1341.295691] Key type ._llcrypt registered [ 1341.298954] Key type .llcrypt registered [ 1341.474277] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1343.156093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.171467] alg: No test for adler32 (adler32-zlib) [ 1344.365688] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1344.741732] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1346.521172] Key type lgssc registered [ 1347.839469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1356.192101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1356.214503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.070193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.509470] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1367.529257] LustreError: 42743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.551143] LustreError: 42743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1372.651289] LustreError: 42744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.002515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.478244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.535608] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1386.360577] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1391.125465] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1398.288031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.863409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.000394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1405.358557] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1411.484603] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.037947] Lustre: server umount lustre-MDT0000 complete [ 1418.567369] LustreError: 42738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.728154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.751535] Lustre: Skipped 1 previous similar message [ 1422.213542] LustreError: 42724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707439 with bad export cookie 10380273657700057036 [ 1422.214073] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.222793] LustreError: 42724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1423.639391] LustreError: 42738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.667638] LustreError: 42738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1423.680741] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 1423.846670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.864477] Lustre: Skipped 1 previous similar message [ 1427.872583] LustreError: 44818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.872874] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1427.895421] LustreError: 44818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1427.897012] Lustre: Skipped 1 previous similar message [ 1428.586422] Lustre: server umount lustre-MDT0001 complete [ 1458.146760] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1458.327694] Lustre: server umount lustre-OST0000 complete [ 1476.109794] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1479.309942] Key type lgssc unregistered [ 1479.619979] LNet: 46249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.636323] LNetError: 46249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.666350] LNet: Removed LNI 192.168.204.148@tcp [ 1480.516222] Key type .llcrypt unregistered [ 1480.519883] Key type ._llcrypt unregistered [ 1509.862489] Key type ._llcrypt registered [ 1509.865029] Key type .llcrypt registered [ 1509.961954] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1511.026319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.083963] alg: No test for adler32 (adler32-zlib) [ 1512.393948] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1512.603757] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1514.266305] Key type lgssc registered [ 1516.087911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1525.616773] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1525.687602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1541.407401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1546.527947] LustreError: 47623:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 1573.280411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.798641] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1587.220682] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1753.319644] Lustre: server umount lustre-OST0000 complete [ 1770.801410] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1773.841673] Key type lgssc unregistered [ 1774.133748] LNet: 49166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.151699] LNetError: 49166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.177496] LNet: Removed LNI 192.168.204.148@tcp [ 1775.013858] Key type .llcrypt unregistered [ 1775.018675] Key type ._llcrypt unregistered [ 1802.251342] Key type ._llcrypt registered [ 1802.256450] Key type .llcrypt registered [ 1802.371956] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1803.831518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1803.912496] alg: No test for adler32 (adler32-zlib) [ 1805.102653] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1805.435907] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1807.175665] Key type lgssc registered [ 1808.468126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.643421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1818.697287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.409517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.507586] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1830.881625] LustreError: 50565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.925383] LustreError: 50565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1835.137700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.638826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.201157] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1846.953186] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1851.493850] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1858.435042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.874717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1865.078707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1865.788751] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1872.902823] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1875.424278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.429925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.434966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.445545] Lustre: Skipped 1 previous similar message [ 1875.461032] Lustre: Skipped 1 previous similar message [ 1880.555438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.564803] Lustre: Skipped 1 previous similar message [ 1881.502411] Lustre: server umount lustre-OST0000 complete [ 1886.689763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1886.712644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.748032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.788047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.793128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.823640] Lustre: Skipped 1 previous similar message [ 1892.557939] Lustre: server umount lustre-MDT0000 complete [ 1895.908236] LustreError: 50564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.926692] LustreError: 50564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1896.758115] LustreError: 50544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779707913 with bad export cookie 16450088519431348771 [ 1896.765578] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.778848] LustreError: 50544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1897.310295] Lustre: server umount lustre-MDT0001 complete [ 1911.427881] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1914.911084] Key type lgssc unregistered [ 1915.277711] LNet: 53742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.290650] LNetError: 53742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.309527] LNet: Removed LNI 192.168.204.148@tcp [ 1916.263325] Key type .llcrypt unregistered [ 1916.268608] Key type ._llcrypt unregistered [ 1942.726445] Key type ._llcrypt registered [ 1942.729488] Key type .llcrypt registered [ 1942.863090] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1944.362031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.419043] alg: No test for adler32 (adler32-zlib) [ 1945.546559] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 1945.878547] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1947.583229] Key type lgssc registered [ 1949.073098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1957.581399] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1957.600044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1973.215499] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1978.335240] LustreError: 55113:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 2005.025064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.181827] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2017.482475] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2024.518866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.990870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.043424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2030.049540] LustreError: 55139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.096322] LustreError: 55139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2030.240392] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2035.170475] LustreError: 55138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.216384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.273859] LustreError: 56036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.295797] LustreError: 56036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2039.749214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.546530] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2050.943298] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2055.446731] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2060.268418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.271449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2060.298648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2063.672678] Lustre: server umount lustre-OST0000 complete [ 2085.856203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2085.862527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.870197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.873687] Lustre: Skipped 2 previous similar messages [ 2085.884952] Lustre: Skipped 2 previous similar messages [ 2090.977149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.991970] Lustre: Skipped 1 previous similar message [ 2091.388156] Lustre: server umount lustre-MDT0000 complete [ 2095.443018] LustreError: 55121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708112 with bad export cookie 6240537306669582667 [ 2095.444135] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2095.458621] LustreError: 55121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2101.968788] Lustre: server umount lustre-MDT0001 complete [ 2113.474402] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2116.285443] Key type lgssc unregistered [ 2116.720564] LNet: 58501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.735831] LNetError: 58501:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.777926] LNet: Removed LNI 192.168.204.148@tcp [ 2117.751451] Key type .llcrypt unregistered [ 2117.755756] Key type ._llcrypt unregistered [ 2140.366330] Key type ._llcrypt registered [ 2140.370166] Key type .llcrypt registered [ 2140.479815] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2141.723209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.788531] alg: No test for adler32 (adler32-zlib) [ 2142.856289] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2143.099765] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2144.775218] Key type lgssc registered [ 2145.954790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2153.381584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2153.414673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.950275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.639773] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2164.199951] LustreError: 59896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.215847] LustreError: 59896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2166.969028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.361809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.265300] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2176.699356] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2180.539598] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2187.381311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2187.710856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.148202] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2199.951843] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2199.998940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2218.470446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.478473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.492257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.472763] Lustre: server umount lustre-OST0000 complete [ 2225.634664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.651519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.660860] Lustre: Skipped 1 previous similar message [ 2230.755939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.763431] Lustre: Skipped 1 previous similar message [ 2234.898302] Lustre: server umount lustre-MDT0000 complete [ 2235.884704] LustreError: 59896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.909667] LustreError: 59896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2238.367566] LustreError: 59876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708255 with bad export cookie 467554204585669506 [ 2238.370693] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.387331] LustreError: 59876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2238.760748] Lustre: server umount lustre-MDT0001 complete [ 2250.543940] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2253.693920] Key type lgssc unregistered [ 2253.977497] LNet: 63094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.989352] LNetError: 63094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.017118] LNet: Removed LNI 192.168.204.148@tcp [ 2254.905376] Key type .llcrypt unregistered [ 2254.909846] Key type ._llcrypt unregistered [ 2280.668455] Key type ._llcrypt registered [ 2280.672052] Key type .llcrypt registered [ 2280.803289] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2281.941773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.958143] alg: No test for adler32 (adler32-zlib) [ 2283.052095] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2283.233943] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2284.880802] Key type lgssc registered [ 2286.000296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.079248] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:25:11 (1779708311) [ 2302.104948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2302.125558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.758767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.989690] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2313.185879] LustreError: 64782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.209427] LustreError: 64782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2318.137611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.308566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2318.544155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2323.571386] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2330.424653] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2334.872548] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2342.076656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.486100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.689369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2350.869690] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2359.591791] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2367.704678] Lustre: Failing over lustre-MDT0000 [ 2368.139145] Lustre: server umount lustre-MDT0000 complete [ 2368.334020] LustreError: 64779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.016829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.047766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.086866] LustreError: 64782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.552236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.562286] Lustre: Skipped 1 previous similar message [ 2373.404556] LustreError: 64777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.428888] LustreError: 64777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2378.533612] LustreError: 64779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.567156] LustreError: 64779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2379.394157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.568847] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.782863] Lustre: *** cfs_fail_loc=135, val=0*** [ 2379.910726] LustreError: 67533:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2379.918475] LustreError: 67533:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -2 [ 2379.940527] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2379.940527] [ 2379.972610] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2379.998806] LustreError: 67521:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2380.025752] LustreError: 67521:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2380.032628] LustreError: 67521:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2380.038720] LustreError: 67521:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2380.085524] Lustre: server umount lustre-MDT0000 complete [ 2380.087752] LustreError: 67521:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2388.761593] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.773735] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2389.144677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.326950] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.342299] LustreError: 64775:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2389.684505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.761522] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2393.892339] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2394.372325] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2395.129448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2395.169836] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2395.216601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2405.345883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.351703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.371445] Lustre: Skipped 1 previous similar message [ 2405.380791] Lustre: Skipped 1 previous similar message [ 2409.792545] Lustre: server umount lustre-OST0000 complete [ 2415.584329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.586268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.600618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.609954] Lustre: Skipped 1 previous similar message [ 2420.705055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.715815] Lustre: Skipped 2 previous similar messages [ 2425.826148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.831223] Lustre: Skipped 1 previous similar message [ 2428.383171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2428.604691] Lustre: server umount lustre-MDT0000 complete [ 2430.948359] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.961677] LustreError: 64778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2432.866462] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708449 with bad export cookie 15466289059165362859 [ 2432.878326] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.888648] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2433.486980] Lustre: server umount lustre-MDT0001 complete [ 2445.747952] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2449.337771] Key type lgssc unregistered [ 2449.740754] LNet: 69281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.750613] LNetError: 69281:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.770950] LNet: Removed LNI 192.168.204.148@tcp [ 2450.620971] Key type .llcrypt unregistered [ 2450.622894] Key type ._llcrypt unregistered [ 2476.829209] Key type ._llcrypt registered [ 2476.831688] Key type .llcrypt registered [ 2476.926640] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2477.972861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2478.057672] alg: No test for adler32 (adler32-zlib) [ 2479.096675] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2479.304827] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2480.945542] Key type lgssc registered [ 2482.244129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2489.971478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2489.993888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.616251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2496.015995] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2500.071886] LustreError: 70677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.104145] LustreError: 70677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2501.095928] LustreError: 70678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.212477] LustreError: 70677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.217897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.693731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.705565] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2520.005666] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2525.176805] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2533.554358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2534.179675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.259993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2542.133759] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2549.463937] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2560.014642] Lustre: Failing over lustre-MDT0000 [ 2560.324513] Lustre: server umount lustre-MDT0000 complete [ 2562.870515] LustreError: 70673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.049261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.062303] Lustre: Skipped 1 previous similar message [ 2567.954141] LustreError: 70674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.984252] LustreError: 70674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2571.593597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.807921] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.003473] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2572.104095] LustreError: 73428:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2572.110334] LustreError: 73428:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -9 [ 2572.116845] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2572.116845] [ 2572.127109] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2572.143252] LustreError: 73416:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2572.152930] LustreError: 73416:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2572.161150] LustreError: 73416:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2572.171100] LustreError: 73416:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2572.210038] Lustre: server umount lustre-MDT0000 complete [ 2572.218838] LustreError: 73416:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2578.211923] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.230399] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2580.931060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.040496] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.048925] LustreError: 73775:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.204.148@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f9c67d41) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2581.064531] LustreError: 70670:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2581.284336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.322483] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2583.315315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2586.038127] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2586.605039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2586.633102] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2586.681409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2596.845047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.857707] Lustre: Skipped 1 previous similar message [ 2596.876673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.892277] Lustre: Skipped 1 previous similar message [ 2601.960779] Lustre: server umount lustre-OST0000 complete [ 2607.082274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.088314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.101632] Lustre: Skipped 1 previous similar message [ 2607.101993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.107845] Lustre: Skipped 1 previous similar message [ 2612.195776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.205058] Lustre: Skipped 1 previous similar message [ 2616.291676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.300247] Lustre: Skipped 1 previous similar message [ 2620.384910] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2620.683092] Lustre: server umount lustre-MDT0000 complete [ 2621.409093] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.426869] LustreError: 70672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2624.706852] LustreError: 70658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708641 with bad export cookie 5236668355311501014 [ 2624.721889] LustreError: 70658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2624.723103] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.093173] Lustre: server umount lustre-MDT0001 complete [ 2638.289453] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2641.716554] Key type lgssc unregistered [ 2642.080684] LNet: 75177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.098893] LNetError: 75177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.128344] LNet: Removed LNI 192.168.204.148@tcp [ 2643.199794] Key type .llcrypt unregistered [ 2643.202562] Key type ._llcrypt unregistered [ 2659.047641] Key type ._llcrypt registered [ 2659.052836] Key type .llcrypt registered [ 2659.196573] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2659.580311] Key type .llcrypt unregistered [ 2659.582192] Key type ._llcrypt unregistered [ 2692.172690] Key type ._llcrypt registered [ 2692.180658] Key type .llcrypt registered [ 2692.369088] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2693.861970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2693.944203] alg: No test for adler32 (adler32-zlib) [ 2695.335489] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2695.679754] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2697.367670] Key type lgssc registered [ 2698.913210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2708.324266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2708.338808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.906761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.017205] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2719.210583] LustreError: 77047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.238276] LustreError: 77047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2724.322333] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.477550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.942320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.985072] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2737.677933] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2742.561534] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2750.185168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.439323] LustreError: 78639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.713917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.838316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2757.531886] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2763.766558] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2776.562393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.567505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2776.567719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.580672] Lustre: Skipped 1 previous similar message [ 2776.608655] Lustre: Skipped 1 previous similar message [ 2780.886710] Lustre: server umount lustre-OST0000 complete [ 2786.792489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.800319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.811268] Lustre: Skipped 1 previous similar message [ 2791.685494] Lustre: server umount lustre-MDT0000 complete [ 2791.921532] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.943702] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2796.075585] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.100937] LustreError: 77046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2796.125069] LustreError: 77027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708813 with bad export cookie 10086149735092929144 [ 2796.154767] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.504961] Lustre: server umount lustre-MDT0001 complete [ 2808.583641] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2811.985810] Key type lgssc unregistered [ 2812.350441] LNet: 80225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2812.360933] LNetError: 80225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2812.377985] LNet: Removed LNI 192.168.204.148@tcp [ 2813.333558] Key type .llcrypt unregistered [ 2813.343582] Key type ._llcrypt unregistered [ 2840.763509] Key type ._llcrypt registered [ 2840.771276] Key type .llcrypt registered [ 2840.914409] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2842.372435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2842.433413] alg: No test for adler32 (adler32-zlib) [ 2843.610294] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2843.916222] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2845.664821] Key type lgssc registered [ 2847.481877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2856.539395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2856.570567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.106022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.741205] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2868.708588] LustreError: 81622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.719611] LustreError: 81622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2873.090217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.483618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.560878] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2885.524410] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2890.174199] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2898.226191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.605325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.773561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2906.331036] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2913.252825] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2923.151986] Lustre: server umount lustre-OST0000 complete [ 2929.633161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2929.649437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2929.660129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.656674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.657233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.682414] Lustre: Skipped 1 previous similar message [ 2935.778867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.792575] Lustre: Skipped 1 previous similar message [ 2937.934109] Lustre: server umount lustre-MDT0000 complete [ 2940.896778] LustreError: 81622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.946129] LustreError: 81622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2941.919306] Lustre: 80933:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779708942/real 1779708942] req@00000000595ea59b x1866159997609600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779708958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2941.920545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.956483] Lustre: 80933:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2942.010763] LustreError: 81618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.033947] LustreError: 81618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2942.389180] LustreError: 81603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779708959 with bad export cookie 16934792123835191107 [ 2942.392530] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.398216] LustreError: 81603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2942.901449] Lustre: server umount lustre-MDT0001 complete [ 2955.821821] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2958.772291] Key type lgssc unregistered [ 2959.164751] LNet: 84797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2959.175783] LNetError: 84797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2959.190416] LNet: Removed LNI 192.168.204.148@tcp [ 2960.158559] Key type .llcrypt unregistered [ 2960.163147] Key type ._llcrypt unregistered [ 2983.805288] Key type ._llcrypt registered [ 2983.809074] Key type .llcrypt registered [ 2983.954045] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2985.191130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2985.294856] alg: No test for adler32 (adler32-zlib) [ 2986.350919] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 2986.625659] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2988.351767] Key type lgssc registered [ 2989.487700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2998.440766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2998.456491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.920577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.135967] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3009.577047] LustreError: 86193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.602264] LustreError: 86193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3013.938779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.452328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.429583] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3025.921414] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3030.091286] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3037.009267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3037.369984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.103379] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3044.547371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3049.834908] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3062.763630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3062.776983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.809289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.316955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.322554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3068.960270] Lustre: server umount lustre-OST0000 complete [ 3075.554659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.571772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.584880] Lustre: Skipped 1 previous similar message [ 3075.601517] Lustre: Skipped 2 previous similar messages [ 3079.512289] Lustre: server umount lustre-MDT0000 complete [ 3080.687846] LustreError: 87157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.716277] LustreError: 87157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3083.946398] LustreError: 86174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709100 with bad export cookie 8698106475191545971 [ 3083.953637] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.956699] LustreError: 86174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3084.405967] Lustre: server umount lustre-MDT0001 complete [ 3095.602235] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3098.704566] Key type lgssc unregistered [ 3099.050662] LNet: 89387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3099.060380] LNetError: 89387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3099.075966] LNet: Removed LNI 192.168.204.148@tcp [ 3099.909832] Key type .llcrypt unregistered [ 3099.921302] Key type ._llcrypt unregistered [ 3122.345248] Key type ._llcrypt registered [ 3122.350674] Key type .llcrypt registered [ 3122.451014] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3123.289787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3123.460539] alg: No test for adler32 (adler32-zlib) [ 3124.488872] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3124.706447] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3126.375222] Key type lgssc registered [ 3127.944422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3137.625790] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3137.704813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.439447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3158.560103] LustreError: 90761:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 3185.314203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.565711] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3197.747778] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3203.582901] Lustre: server umount lustre-OST0000 complete [ 3212.115300] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:40:28 (1779709228) [ 3219.011464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.407394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.185044] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3224.551305] LustreError: 92288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.593914] LustreError: 92288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3229.667473] LustreError: 92287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.379040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.774245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3239.193653] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3244.947651] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3248.697340] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3255.167039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3255.489693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3261.510232] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3264.700109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3266.613468] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3280.360539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3280.365020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.374876] LustreError: Skipped 1 previous similar message [ 3280.388151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3280.398625] Lustre: Skipped 1 previous similar message [ 3282.493613] Lustre: server umount lustre-OST0000 complete [ 3290.594326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.620126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.626030] Lustre: Skipped 2 previous similar messages [ 3290.650778] Lustre: Skipped 1 previous similar message [ 3293.121298] Lustre: server umount lustre-MDT0000 complete [ 3295.715702] LustreError: 94653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.748468] LustreError: 94653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3297.468447] LustreError: 92269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709314 with bad export cookie 4237947201244658624 [ 3297.478674] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.483142] LustreError: 92269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3298.169442] Lustre: server umount lustre-MDT0001 complete [ 3311.033855] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3314.046854] Key type lgssc unregistered [ 3314.330258] LNet: 95621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3314.339564] LNetError: 95621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3315.374302] LNet: Removed LNI 192.168.204.148@tcp [ 3316.119640] Key type .llcrypt unregistered [ 3316.122122] Key type ._llcrypt unregistered [ 3358.051542] Key type ._llcrypt registered [ 3358.055059] Key type .llcrypt registered [ 3358.184712] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3359.381518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3359.411981] alg: No test for adler32 (adler32-zlib) [ 3360.437800] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3360.752803] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3362.437697] Key type lgssc registered [ 3363.413889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.566522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.584342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.105986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.493465] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3382.248939] LustreError: 97793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.276210] LustreError: 97793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3384.263774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.587097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.665024] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3394.616253] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.361599] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3405.928434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.350696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.125407] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3417.653054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3418.846719] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3427.813414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3427.828493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.853101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3427.857070] Lustre: Skipped 1 previous similar message [ 3432.934099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3432.944272] Lustre: Skipped 1 previous similar message [ 3433.823453] Lustre: server umount lustre-OST0000 complete [ 3438.051588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.058230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.084863] Lustre: Skipped 2 previous similar messages [ 3443.170594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3443.177171] Lustre: Skipped 2 previous similar messages [ 3448.292606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.296209] Lustre: Skipped 1 previous similar message [ 3452.384608] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.774883] Lustre: server umount lustre-MDT0000 complete [ 3453.408880] LustreError: 98504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.419658] LustreError: 98504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3457.660981] LustreError: 97776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709474 with bad export cookie 12975196226279046992 [ 3457.674519] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3457.676923] LustreError: 97776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3458.162966] Lustre: server umount lustre-MDT0001 complete [ 3469.686787] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3472.334261] Key type lgssc unregistered [ 3472.733663] LNet: 100981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3472.746270] LNetError: 100981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3472.777416] LNet: Removed LNI 192.168.204.148@tcp [ 3473.903468] Key type .llcrypt unregistered [ 3473.906082] Key type ._llcrypt unregistered [ 3492.188522] Key type ._llcrypt registered [ 3492.190955] Key type .llcrypt registered [ 3492.285328] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3493.621348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3493.676742] alg: No test for adler32 (adler32-zlib) [ 3494.821167] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 3495.094718] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3496.783178] Key type lgssc registered [ 3498.242533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3507.082467] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3507.121124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.720137] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3527.840131] LustreError: 102209:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 3554.655195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.079817] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3566.520537] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3573.086384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.309082] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3573.316611] LustreError: 103120:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3573.321574] LustreError: 103120:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3573.329548] LustreError: 103120:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3573.390441] Lustre: server umount lustre-MDT0000 complete [ 3573.392642] LustreError: 103120:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3594.720000] Lustre: 102222:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779709595/real 1779709595] req@00000000f2e19cec x1866160681258368/t0(0) o101->MGC192.168.204.148@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1779709611 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3594.721484] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3594.757403] Lustre: 102222:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3594.771553] LustreError: 102222:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3615.281701] Lustre: server umount lustre-OST0000 complete [ 3627.329989] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3638.930790] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3682.735060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.182226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3683.238529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3683.353688] Lustre: lustre-MDT0000: new disk, initializing [ 3683.453783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3683.477495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3688.336225] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3699.964302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.100226] Lustre: 105881:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3700.172226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3700.175450] Lustre: Skipped 1 previous similar message [ 3700.251222] Lustre: lustre-MDT0001: new disk, initializing [ 3700.348235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3700.390185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3700.408212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3704.812893] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3713.400248] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3718.007572] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3726.076706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3726.365785] Lustre: lustre-OST0000: new disk, initializing [ 3726.371704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3726.379301] Lustre: 107620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3726.467537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.893279] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3733.615891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3733.636848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3733.820234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3741.595992] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3746.492939] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3746.827663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3751.488362] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3751.778909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3754.465323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.468795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.472387] LustreError: Skipped 1 previous similar message [ 3754.476535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3754.482434] Lustre: Skipped 1 previous similar message [ 3754.503881] Lustre: Skipped 1 previous similar message [ 3759.605394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.622504] Lustre: Skipped 1 previous similar message [ 3764.708557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.723947] Lustre: Skipped 1 previous similar message [ 3768.287272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3768.707911] Lustre: server umount lustre-OST0000 complete [ 3774.948469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.969423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.978933] Lustre: Skipped 1 previous similar message [ 3779.775688] Lustre: server umount lustre-MDT0000 complete [ 3783.885723] LustreError: 105874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709800 with bad export cookie 4741681949346967616 [ 3783.899663] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.902219] LustreError: 105874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3784.231720] Lustre: server umount lustre-MDT0001 complete [ 3794.084934] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:50:10 (1779709810) [ 3797.771271] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3815.167591] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3824.594701] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3862.171936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3862.414307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3862.439927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3862.538716] Lustre: lustre-MDT0000: new disk, initializing [ 3862.626313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.655779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3867.938918] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3879.032551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.102521] Lustre: 112348:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3879.198876] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3879.202200] Lustre: Skipped 1 previous similar message [ 3879.290142] Lustre: lustre-MDT0001: new disk, initializing [ 3879.397087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3879.434168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3879.447290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3884.599117] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3892.716644] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3897.268949] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3905.806794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3906.211138] Lustre: lustre-OST0000: new disk, initializing [ 3906.222849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3906.234589] Lustre: 114091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3906.359736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.929733] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3914.349088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3914.366847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3914.543847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3922.203519] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3927.830088] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3928.103712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3932.542341] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3932.792925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3935.200532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3935.200633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.209218] LustreError: Skipped 1 previous similar message [ 3935.210202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.225239] Lustre: Skipped 2 previous similar messages [ 3935.236416] Lustre: Skipped 1 previous similar message [ 3940.772529] Lustre: server umount lustre-OST0000 complete [ 3945.447318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.450559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3945.471609] Lustre: Skipped 1 previous similar message [ 3945.480979] Lustre: Skipped 3 previous similar messages [ 3950.880727] Lustre: server umount lustre-MDT0000 complete [ 3954.992389] LustreError: 112340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779709971 with bad export cookie 4741681949346968505 [ 3954.997226] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.000365] LustreError: 112340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3955.337955] Lustre: server umount lustre-MDT0001 complete [ 3965.142540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.701993] LustreError: 116160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.730453] LustreError: 116160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3965.778938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.961264] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3971.044149] LustreError: 116161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.141032] LustreError: 116161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.264064] LustreError: 116160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3981.049557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.733735] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3992.908582] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3997.135561] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4004.590270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.799550] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4019.764420] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4028.268599] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4036.065436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.082574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4036.090763] Lustre: Skipped 3 previous similar messages [ 4039.169901] Lustre: server umount lustre-OST0000 complete [ 4046.308885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.329621] Lustre: Skipped 2 previous similar messages [ 4049.530929] Lustre: server umount lustre-MDT0000 complete [ 4051.430216] LustreError: 116161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.477045] LustreError: 116161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4053.859089] LustreError: 118700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710070 with bad export cookie 4741681949346969387 [ 4053.859422] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4053.870777] LustreError: 118700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4054.296075] Lustre: server umount lustre-MDT0001 complete [ 4065.431624] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4068.234420] Key type lgssc unregistered [ 4068.491587] LNet: 119436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4068.497759] LNetError: 119436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4068.509781] LNet: Removed LNI 192.168.204.148@tcp [ 4069.146240] Key type .llcrypt unregistered [ 4069.149393] Key type ._llcrypt unregistered [ 4087.128848] Key type ._llcrypt registered [ 4087.130886] Key type .llcrypt registered [ 4087.245494] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4106.017423] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4107.946564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4108.137523] alg: No test for adler32 (adler32-zlib) [ 4109.557636] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 4109.952338] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4111.687189] Key type lgssc registered [ 4113.288189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4155.453281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4155.484910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.777821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4156.825218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4156.958479] Lustre: lustre-MDT0000: new disk, initializing [ 4157.100610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.122901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4161.639451] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4175.883822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.040803] Lustre: 122559:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4176.093193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4176.097744] Lustre: Skipped 1 previous similar message [ 4176.183941] Lustre: lustre-MDT0001: new disk, initializing [ 4176.441856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.503663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4176.519439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4181.971248] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4191.696487] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4196.155530] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4203.687281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.986577] Lustre: lustre-OST0000: new disk, initializing [ 4203.989822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4203.995961] Lustre: 124309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4204.094293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.403421] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4211.806096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4211.819804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4211.956320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4219.814302] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4225.851662] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4226.124970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4231.698873] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4232.025365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4237.792929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4237.799772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.804830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4237.816252] Lustre: Skipped 1 previous similar message [ 4237.833461] Lustre: Skipped 1 previous similar message [ 4240.552824] Lustre: server umount lustre-OST0000 complete [ 4248.042885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.043240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4248.051964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.072552] Lustre: Skipped 1 previous similar message [ 4248.107626] LustreError: Skipped 1 previous similar message [ 4251.032834] Lustre: server umount lustre-MDT0000 complete [ 4255.032888] LustreError: 122554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710271 with bad export cookie 17570691836127535152 [ 4255.036922] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4255.049425] LustreError: 122554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4255.502566] Lustre: server umount lustre-MDT0001 complete [ 4265.582580] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:58:01 (1779710281) [ 4272.817587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.292718] LustreError: 126574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.306470] LustreError: 126574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4273.380931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.160712] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4278.759797] LustreError: 126575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.884651] LustreError: 126574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.547596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.966382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4294.326753] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4300.881637] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4305.505744] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4308.388591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.397540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.401429] Lustre: Skipped 2 previous similar messages [ 4313.506360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.525948] Lustre: Skipped 1 previous similar message [ 4314.845402] Lustre: server umount lustre-MDT0000 complete [ 4318.706283] LustreError: 127510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.739113] LustreError: 127510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4319.983808] LustreError: 126556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710336 with bad export cookie 17570691836127536034 [ 4319.991815] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4319.994018] LustreError: 126556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4320.368389] Lustre: server umount lustre-MDT0001 complete [ 4329.725882] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:59:05 (1779710345) [ 4336.414920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4351.904378] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4357.023804] LustreError: 128763:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 4382.751302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.108490] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4394.973830] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4397.006765] Lustre: server umount lustre-OST0000 complete [ 4406.253413] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:00:22 (1779710422) [ 4414.654037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.192896] LustreError: 130096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.337093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.131212] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4425.705843] LustreError: 130097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.718924] LustreError: 130097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4429.510607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.795521] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4441.095598] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4446.105054] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4454.625453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.821200] LustreError: 131704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.024816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.033117] Lustre: Skipped 1 previous similar message [ 4461.700800] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4467.876221] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4481.001673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4481.007803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.017968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4481.032255] Lustre: Skipped 3 previous similar messages [ 4481.046360] Lustre: Skipped 2 previous similar messages [ 4482.417724] Lustre: server umount lustre-MDT0000 complete [ 4486.827944] LustreError: 130078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710503 with bad export cookie 17570691836127536699 [ 4486.838091] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4486.841231] LustreError: 130078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4487.390940] Lustre: server umount lustre-MDT0001 complete [ 4502.040452] Lustre: server umount lustre-OST0000 complete [ 4512.355680] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:02:07 (1779710527) [ 4521.794349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.906739] LustreError: 133427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4522.943040] LustreError: 133427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4523.047580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.650308] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4539.627836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.513983] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4553.218584] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4558.389654] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4566.401484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4567.997588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4574.997985] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4581.019312] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4585.938725] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4586.203924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4591.887664] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4592.085281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4595.681215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.685046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.696401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.701733] Lustre: Skipped 1 previous similar message [ 4600.619685] Lustre: server umount lustre-OST0000 complete [ 4605.922885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.935227] Lustre: Skipped 1 previous similar message [ 4611.440229] Lustre: server umount lustre-MDT0000 complete [ 4615.139321] LustreError: 134539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.160481] LustreError: 134539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4615.293758] LustreError: 133407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710632 with bad export cookie 17570691836127537763 [ 4615.299131] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.301906] LustreError: 133407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4615.654444] Lustre: server umount lustre-MDT0001 complete [ 4625.206640] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:04:01 (1779710641) [ 4632.600326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.160678] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4653.282126] LustreError: 137216:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 4679.137871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4679.143894] Lustre: Skipped 2 previous similar messages [ 4685.438328] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4692.137929] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4699.598748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.385170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4705.115571] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4715.065624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.497980] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4726.950577] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4731.761726] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4736.858942] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4737.075569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4741.944476] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4742.176208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4746.210494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.222882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4746.231114] Lustre: Skipped 1 previous similar message [ 4746.241019] Lustre: Skipped 5 previous similar messages [ 4751.024264] Lustre: server umount lustre-MDT0000 complete [ 4751.342217] LustreError: 137242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.357068] LustreError: 137242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4755.012748] LustreError: 137224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710771 with bad export cookie 17570691836127538792 [ 4755.028521] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.034809] LustreError: 137224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4755.346170] Lustre: server umount lustre-MDT0001 complete [ 4769.388056] Lustre: server umount lustre-OST0000 complete [ 4779.152858] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 08:06:34 (1779710794) [ 4787.829115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4803.424246] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4808.546652] LustreError: 141039:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 4841.170413] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4848.234098] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4856.189457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.522855] LustreError: 120778:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d2080af9 x1866161325234560/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4857.085283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4862.359903] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4873.974654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.268115] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4886.171737] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4891.237723] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4900.070572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4900.208271] Lustre: lustre-OST0001: new disk, initializing [ 4900.217286] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4900.243155] Lustre: 143575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4907.584180] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4910.676949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4910.683994] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4910.867132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4917.067976] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4923.255420] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4923.496678] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4928.622229] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4928.928273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4931.568880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.580069] Lustre: Skipped 2 previous similar messages [ 4931.580160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4931.596087] Lustre: Skipped 1 previous similar message [ 4937.732469] Lustre: server umount lustre-OST0000 complete [ 4946.917954] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.829188] Lustre: server umount lustre-OST0001 complete [ 4956.138521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4956.143897] LustreError: Skipped 1 previous similar message [ 4960.484604] Lustre: server umount lustre-MDT0000 complete [ 4964.855827] LustreError: 141046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779710981 with bad export cookie 17570691836127539744 [ 4964.863260] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4964.880989] LustreError: 141046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4965.347887] Lustre: server umount lustre-MDT0001 complete [ 4994.097660] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 08:10:10 (1779711010) [ 4995.885415] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4997.600809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4999.854785] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 08:10:15 (1779711015) [ 5008.102267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.354620] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5008.385774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5008.667589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.678897] Lustre: Skipped 6 previous similar messages [ 5013.231790] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5022.542427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.618314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5022.641968] Lustre: 146989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5026.993836] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5032.366351] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5036.579548] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5043.032812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.233787] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5049.492395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5050.188285] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5056.621399] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5061.291477] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5061.502715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5066.366740] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5066.580473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5069.807582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.820298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.837528] Lustre: Skipped 4 previous similar messages [ 5069.848788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5069.857105] Lustre: Skipped 7 previous similar messages [ 5074.724832] Lustre: server umount lustre-OST0000 complete [ 5097.054551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.093957] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5109.331798] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5110.708411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.724604] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5115.026167] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5115.260488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5120.667657] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5120.952586] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.967068] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5121.977262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 5126.004613] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a304a329800.ost_server_uuid 50 [ 5127.567536] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a304a329800.ost_server_uuid in FULL state after 0 sec [ 5127.860108] LustreError: 147803:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.48@tcp) returned error from blocking AST (req@000000002cd95780 x1866161325341440 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c79b0b6d/0xf3d7a321a63cfb7a lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.48@tcp remote: 0x8d67e568f03dc9bc expref: 5 pid: 147803 timeout: 5227 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5127.901619] LustreError: lustre-MDT0000: A client on nid 192.168.204.48@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5127.916388] LustreError: 146986:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.48@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c79b0b6d/0xf3d7a321a63cfb7a lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.48@tcp remote: 0x8d67e568f03dc9bc expref: 6 pid: 147803 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5133.805344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5133.813050] LustreError: Skipped 1 previous similar message [ 5138.981391] Lustre: server umount lustre-OST0000 complete [ 5157.346404] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5157.712639] Lustre: server umount lustre-MDT0000 complete [ 5161.680642] LustreError: 146983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711178 with bad export cookie 17570691836127540927 [ 5161.689220] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.068050] Lustre: server umount lustre-MDT0001 complete [ 5181.682732] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5184.377524] Key type lgssc unregistered [ 5184.705982] LNet: 152480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5184.712284] LNetError: 152480:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5184.732177] LNet: Removed LNI 192.168.204.148@tcp [ 5185.534386] Key type .llcrypt unregistered [ 5185.542137] Key type ._llcrypt unregistered [ 5210.742834] Key type ._llcrypt registered [ 5210.746226] Key type .llcrypt registered [ 5211.003607] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5212.313844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5212.380755] alg: No test for adler32 (adler32-zlib) [ 5213.676134] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5214.068649] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5215.759206] Key type lgssc registered [ 5217.135870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5225.696530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5225.737290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.426898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.890399] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5237.730706] LustreError: 153876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.756592] LustreError: 153876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5241.371173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.755242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5247.480287] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5254.999907] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5260.421968] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5269.452397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5269.934506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.030862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5277.020804] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5283.585508] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5294.033305] Lustre: Failing over lustre-MDT0000 [ 5294.405468] Lustre: server umount lustre-MDT0000 complete [ 5297.635079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.653146] LustreError: 156026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.668862] Lustre: Skipped 2 previous similar messages [ 5297.694940] LustreError: 156026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5302.764306] LustreError: 153872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.794850] LustreError: 153872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5307.001786] LustreError: 153855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.48@tcp arrived at 1779711323 with bad export cookie 9178773439604972202 [ 5307.877640] LustreError: 153876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.895236] LustreError: 153876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5312.998743] LustreError: 153875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5313.015714] LustreError: 153875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5314.015228] Lustre: 153187:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779711314/real 1779711314] req@000000001e846305 x1866162483784064/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779711330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5314.051825] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5315.652960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.278063] LustreError: 153872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.302888] LustreError: 153872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5324.637555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.707565] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5329.781617] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5329.904335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5329.912980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5332.510577] Lustre: lustre-MDT0000: Denying connection for new client c5ebe87e-1237-426c-961e-2f4addbd1da2 (at 192.168.204.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5343.713434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.718983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.751823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5345.253650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5345.261652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.038173] Lustre: server umount lustre-OST0000 complete [ 5351.204941] LustreError: 157247:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5351.213940] Lustre: 156613:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5351.218655] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5351.229975] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5351.238099] LustreError: 156613:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004c6cf5fb x1866162483804672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5351.249636] LustreError: 156613:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5351.256503] LustreError: 156613:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5351.268540] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5351.556333] Lustre: server umount lustre-MDT0000 complete [ 5355.293637] LustreError: 153856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711372 with bad export cookie 9178773439604972307 [ 5355.303454] LustreError: 153185:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5355.314138] LustreError: 153856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5355.320097] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.675581] Lustre: server umount lustre-MDT0001 complete [ 5371.720475] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5374.239591] Key type lgssc unregistered [ 5374.517729] LNet: 157984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5374.525293] LNetError: 157984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5374.560062] LNet: Removed LNI 192.168.204.148@tcp [ 5375.239734] Key type .llcrypt unregistered [ 5375.242181] Key type ._llcrypt unregistered [ 5399.944497] Key type ._llcrypt registered [ 5399.947462] Key type .llcrypt registered [ 5400.122701] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5401.659929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5401.686131] alg: No test for adler32 (adler32-zlib) [ 5402.909403] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5403.234163] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5404.983275] Key type lgssc registered [ 5406.652548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5416.777312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5416.818858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.431560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.739292] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5428.724638] LustreError: 159383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.736293] LustreError: 159383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5433.322689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.817933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5438.443317] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5444.567385] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5448.181946] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5456.026104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.538756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5463.757114] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5464.846500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5470.057689] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5480.434660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5480.437769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.445461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5480.461411] Lustre: Skipped 1 previous similar message [ 5483.279984] Lustre: server umount lustre-OST0000 complete [ 5490.146603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5490.158894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.187626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.190495] Lustre: Skipped 1 previous similar message [ 5494.475449] Lustre: server umount lustre-MDT0000 complete [ 5495.780449] LustreError: 159382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.797562] LustreError: 159382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5498.419861] LustreError: 159365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711515 with bad export cookie 9713901896080697030 [ 5498.440971] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5498.849371] Lustre: server umount lustre-MDT0001 complete [ 5509.884723] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5513.021634] Key type lgssc unregistered [ 5513.293952] LNet: 162557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5513.301872] LNetError: 162557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.324855] LNet: Removed LNI 192.168.204.148@tcp [ 5514.143919] Key type .llcrypt unregistered [ 5514.148429] Key type ._llcrypt unregistered [ 5549.722727] Key type ._llcrypt registered [ 5549.724886] Key type .llcrypt registered [ 5549.820800] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5550.980327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5551.021728] alg: No test for adler32 (adler32-zlib) [ 5552.126074] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5552.386849] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5554.063172] Key type lgssc registered [ 5555.103109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5562.101583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5562.127866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.623486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.854695] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5573.091660] LustreError: 164241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.121726] LustreError: 164241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5578.036848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.211568] LustreError: 164242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.791563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5583.670461] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5590.368578] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5594.877132] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5602.813724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.267742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.377247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5609.983226] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5616.427557] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5624.800859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.818367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.846265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5626.618853] Lustre: server umount lustre-OST0000 complete [ 5634.016810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.041355] Lustre: Skipped 1 previous similar message [ 5634.041567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.068024] Lustre: Skipped 1 previous similar message [ 5639.138675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5639.151134] Lustre: Skipped 1 previous similar message [ 5643.234052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5643.239633] Lustre: Skipped 1 previous similar message [ 5645.281036] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.517724] Lustre: server umount lustre-MDT0000 complete [ 5648.361474] LustreError: 164236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5648.388243] LustreError: 164236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5649.528829] LustreError: 164222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711666 with bad export cookie 14797167055982076498 [ 5649.537993] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5649.542563] LustreError: 164222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5649.916360] Lustre: server umount lustre-MDT0001 complete [ 5662.429903] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5665.642767] Key type lgssc unregistered [ 5665.887646] LNet: 167416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5665.897407] LNetError: 167416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5665.917217] LNet: Removed LNI 192.168.204.148@tcp [ 5666.863511] Key type .llcrypt unregistered [ 5666.865085] Key type ._llcrypt unregistered [ 5689.581541] Key type ._llcrypt registered [ 5689.584522] Key type .llcrypt registered [ 5689.726595] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5690.837450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5690.938312] alg: No test for adler32 (adler32-zlib) [ 5692.043796] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5692.358628] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5694.103235] Key type lgssc registered [ 5695.397456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5708.348766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5708.367396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.999354] Lustre: *** cfs_fail_loc=135, val=0*** [ 5710.111060] LustreError: 168925:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5710.127963] LustreError: 168925:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -2 [ 5710.142563] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5710.142563] [ 5710.153516] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5710.159211] LustreError: 168886:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5710.163342] LustreError: 168886:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5710.166878] LustreError: 168886:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5710.171903] LustreError: 168886:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5710.228224] Lustre: server umount lustre-MDT0000 complete [ 5710.230619] LustreError: 168886:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5718.465856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.065331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.606861] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5728.226578] LustreError: 169296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5728.245628] LustreError: 169296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5733.351980] LustreError: 169297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.232912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.605275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.699911] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5745.899220] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5750.224271] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5752.182793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.208489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.214985] Lustre: Skipped 1 previous similar message [ 5757.408446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.424878] Lustre: Skipped 1 previous similar message [ 5758.646250] Lustre: server umount lustre-MDT0000 complete [ 5759.465842] LustreError: 169297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.481701] LustreError: 169297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5762.492639] LustreError: 169277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779711779 with bad export cookie 8122381319718702817 [ 5762.498579] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.502430] LustreError: 169277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5763.077808] Lustre: server umount lustre-MDT0001 complete [ 5775.603391] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5779.223924] Key type lgssc unregistered [ 5779.669277] LNet: 171378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5779.681082] LNetError: 171378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5779.708925] LNet: Removed LNI 192.168.204.148@tcp [ 5780.745219] Key type .llcrypt unregistered [ 5780.747360] Key type ._llcrypt unregistered [ 5809.028839] Key type ._llcrypt registered [ 5809.033064] Key type .llcrypt registered [ 5809.221851] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5809.660552] Key type .llcrypt unregistered [ 5809.664106] Key type ._llcrypt unregistered [ 5829.846264] Key type ._llcrypt registered [ 5829.851826] Key type .llcrypt registered [ 5830.033352] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5831.716329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5831.842993] alg: No test for adler32 (adler32-zlib) [ 5833.043557] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 5833.317832] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5835.007202] Key type lgssc registered [ 5836.684791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5846.948781] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5846.986505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5862.559459] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5867.680624] LustreError: 173486:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 5894.431592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.714366] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5909.132026] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5919.363285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.148100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.228591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5925.358627] LustreError: 173512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.386344] LustreError: 173512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5925.906564] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5929.449024] LustreError: 174411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.475430] LustreError: 174411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5934.573194] LustreError: 173511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.592528] LustreError: 173511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5936.963568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.483296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.037189] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5949.966634] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5954.940736] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5958.520560] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5971.429477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5971.439057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.452741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5973.474731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.486706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.490274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5973.504183] Lustre: Skipped 1 previous similar message [ 5978.596613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.613541] Lustre: Skipped 1 previous similar message [ 5983.723183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.735550] Lustre: Skipped 1 previous similar message [ 5985.248060] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5985.384974] Lustre: server umount lustre-OST0000 complete [ 5993.959544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.969982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5993.981106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.997107] Lustre: Skipped 1 previous similar message [ 5997.218691] Lustre: server umount lustre-MDT0000 complete [ 5999.087924] LustreError: 174414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.142759] LustreError: 174414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6003.033725] LustreError: 173494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712019 with bad export cookie 3970925537496103915 [ 6003.037709] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6003.068386] LustreError: 173494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6003.528699] Lustre: server umount lustre-MDT0001 complete [ 6017.992862] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6021.794468] Key type lgssc unregistered [ 6022.147371] LNet: 177097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6022.165352] LNetError: 177097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6022.192150] LNet: Removed LNI 192.168.204.148@tcp [ 6023.254769] Key type .llcrypt unregistered [ 6023.256899] Key type ._llcrypt unregistered [ 6049.487183] Key type ._llcrypt registered [ 6049.495247] Key type .llcrypt registered [ 6049.617919] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6051.076820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6051.139452] alg: No test for adler32 (adler32-zlib) [ 6052.190131] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6052.483331] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6054.143183] Key type lgssc registered [ 6055.535296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6065.223297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6065.267278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.860662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.297548] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6076.452318] LustreError: 178491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.481855] LustreError: 178491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6081.506478] LustreError: 178492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.599860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.062597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6086.959927] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6093.218949] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6097.490446] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6105.815325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.223447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.360223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6112.947616] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6120.588724] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6126.157375] Lustre: Failing over lustre-MDT0000 [ 6126.398278] Lustre: server umount lustre-MDT0000 complete [ 6127.075669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.084131] LustreError: 179903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6127.098229] Lustre: Skipped 1 previous similar message [ 6127.140896] LustreError: 179903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6132.194481] LustreError: 180152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.213305] LustreError: 180152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6137.323105] LustreError: 178486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.350631] LustreError: 178486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6143.327166] Lustre: 177803:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779712144/real 1779712144] req@000000005c0d8126 x1866163362489728/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779712160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6143.355895] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6145.723574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.387963] LustreError: 178486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.401310] LustreError: 178486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6153.700580] LustreError: 177801:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004534f81c x1866163362495872/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6154.095476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6154.174195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6158.627685] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6158.846738] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6159.350871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6159.382564] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6159.464373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6164.190380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6168.561891] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6184.421650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6184.433641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6184.452564] Lustre: Skipped 1 previous similar message [ 6184.457625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.680830] Lustre: server umount lustre-OST0000 complete [ 6195.169769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.180359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6195.192399] Lustre: Skipped 1 previous similar message [ 6195.194287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.194293] Lustre: Skipped 1 previous similar message [ 6200.295322] LustreError: 178491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.314629] LustreError: 178491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6200.437497] Lustre: server umount lustre-MDT0000 complete [ 6204.526080] LustreError: 180113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712221 with bad export cookie 8926541047536179647 [ 6204.527545] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6204.537106] LustreError: 180113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6205.001992] Lustre: server umount lustre-MDT0001 complete [ 6216.646631] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6219.755449] Key type lgssc unregistered [ 6220.033702] LNet: 182933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6220.044576] LNetError: 182933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6220.069699] LNet: Removed LNI 192.168.204.148@tcp [ 6220.904900] Key type .llcrypt unregistered [ 6220.908074] Key type ._llcrypt unregistered [ 6245.866435] Key type ._llcrypt registered [ 6245.874335] Key type .llcrypt registered [ 6246.031884] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6247.608253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6247.648090] alg: No test for adler32 (adler32-zlib) [ 6248.765959] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6249.044452] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6250.735180] Key type lgssc registered [ 6252.153499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6259.651992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6259.673434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.075823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.438221] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6270.444429] LustreError: 184328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.466241] LustreError: 184328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6274.410437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.755909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6279.338652] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6284.975745] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6288.456888] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6295.744359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.180519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.865908] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6305.414347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6308.617837] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6311.520181] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6320.475757] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6328.523268] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6341.604503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.652678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6341.658747] Lustre: Skipped 1 previous similar message [ 6346.733713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6346.743396] Lustre: Skipped 1 previous similar message [ 6347.587310] Lustre: server umount lustre-OST0000 complete [ 6356.959940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6356.963733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.975897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6356.984273] Lustre: Skipped 2 previous similar messages [ 6358.692438] Lustre: server umount lustre-MDT0000 complete [ 6362.090773] LustreError: 184322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.116525] LustreError: 184322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6362.241724] LustreError: 184309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712379 with bad export cookie 12459099513924356771 [ 6362.249544] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6362.251329] LustreError: 184309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6362.594094] Lustre: server umount lustre-MDT0001 complete [ 6374.467511] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6377.871288] Key type lgssc unregistered [ 6378.220693] LNet: 187686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6378.227822] LNetError: 187686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6379.247170] LNet: Removed LNI 192.168.204.148@tcp [ 6379.957137] Key type .llcrypt unregistered [ 6379.959068] Key type ._llcrypt unregistered [ 6406.546820] Key type ._llcrypt registered [ 6406.551749] Key type .llcrypt registered [ 6406.755185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6408.776425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6408.901718] alg: No test for adler32 (adler32-zlib) [ 6410.185391] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6410.564960] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6412.287265] Key type lgssc registered [ 6413.574310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6424.917205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6424.932679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.745289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.282451] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6436.836073] LustreError: 189082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6436.876506] LustreError: 189082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6441.571220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.954633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6442.204865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6446.385388] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6452.055233] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6455.904600] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6463.033867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6463.390618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.066754] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6471.598936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6474.542929] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6479.755875] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6492.286392] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6502.865587] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6512.578758] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6527.981717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.988775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6528.004366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6528.020272] Lustre: Skipped 1 previous similar message [ 6530.484152] Lustre: server umount lustre-OST0000 complete [ 6538.214050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.239654] Lustre: Skipped 1 previous similar message [ 6538.254356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.271399] Lustre: Skipped 1 previous similar message [ 6540.942811] Lustre: server umount lustre-MDT0000 complete [ 6543.329426] LustreError: 189793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.354216] LustreError: 189793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6545.006278] LustreError: 189062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712561 with bad export cookie 4164487945554504308 [ 6545.009213] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.016286] LustreError: 189062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6545.413273] Lustre: server umount lustre-MDT0001 complete [ 6559.383824] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6562.631889] Key type lgssc unregistered [ 6562.997700] LNet: 193842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6563.012768] LNetError: 193842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6563.035530] LNet: Removed LNI 192.168.204.148@tcp [ 6564.087830] Key type .llcrypt unregistered [ 6564.089665] Key type ._llcrypt unregistered [ 6592.230616] Key type ._llcrypt registered [ 6592.232784] Key type .llcrypt registered [ 6592.353118] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6593.160866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6593.274131] alg: No test for adler32 (adler32-zlib) [ 6594.346667] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6594.553638] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6596.207221] Key type lgssc registered [ 6597.596487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6606.050871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6606.071170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.626433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6611.982600] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6617.061519] LustreError: 195237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.083406] LustreError: 195237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6620.749241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.206820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.217756] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6632.326760] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6635.910646] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6645.067144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6645.813356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.997987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6653.721343] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6660.941581] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6678.498995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.502694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6678.525153] Lustre: Skipped 1 previous similar message [ 6678.538431] Lustre: Skipped 1 previous similar message [ 6683.616560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6683.642284] Lustre: Skipped 1 previous similar message [ 6684.054470] Lustre: server umount lustre-OST0000 complete [ 6692.841539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6692.859197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.895445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6695.787057] Lustre: server umount lustre-MDT0000 complete [ 6699.010164] LustreError: 195238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6699.038621] LustreError: 195238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6701.826995] LustreError: 195217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712718 with bad export cookie 5274792724424701636 [ 6701.851443] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6701.854569] LustreError: 195217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6702.542747] Lustre: server umount lustre-MDT0001 complete [ 6714.366185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6717.916182] Key type lgssc unregistered [ 6718.343992] LNet: 198568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6718.354085] LNetError: 198568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6718.374067] LNet: Removed LNI 192.168.204.148@tcp [ 6719.528793] Key type .llcrypt unregistered [ 6719.531121] Key type ._llcrypt unregistered [ 6743.968114] Key type ._llcrypt registered [ 6743.971204] Key type .llcrypt registered [ 6744.125792] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6745.147265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6745.308719] alg: No test for adler32 (adler32-zlib) [ 6746.479796] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6746.801135] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6748.599925] Key type lgssc registered [ 6749.822655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6757.769184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6757.792433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.202776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6763.396928] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6769.670173] LustreError: 199817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.689717] LustreError: 199817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6772.297733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.737351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6776.932874] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6782.592655] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6787.251653] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6794.801933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.236649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.388731] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6804.473086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6808.537766] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6824.963233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6824.984400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6824.984957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.004747] Lustre: Skipped 1 previous similar message [ 6830.052325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6830.057722] Lustre: Skipped 1 previous similar message [ 6830.298531] Lustre: server umount lustre-OST0000 complete [ 6835.169122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.173843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6835.195909] Lustre: Skipped 2 previous similar messages [ 6835.216583] Lustre: Skipped 1 previous similar message [ 6840.289535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6840.301539] Lustre: Skipped 1 previous similar message [ 6840.514105] Lustre: server umount lustre-MDT0000 complete [ 6844.285122] LustreError: 199799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779712861 with bad export cookie 10819956859622548623 [ 6844.307067] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6844.619811] Lustre: server umount lustre-MDT0001 complete [ 6856.584469] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6859.741680] Key type lgssc unregistered [ 6860.087528] LNet: 203143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6860.100672] LNetError: 203143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6860.121784] LNet: Removed LNI 192.168.204.148@tcp [ 6860.995647] Key type .llcrypt unregistered [ 6861.000196] Key type ._llcrypt unregistered [ 6882.960887] Key type ._llcrypt registered [ 6882.963423] Key type .llcrypt registered [ 6883.087185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6884.290521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6884.312497] alg: No test for adler32 (adler32-zlib) [ 6885.412751] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 6885.596534] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6887.239221] Key type lgssc registered [ 6888.599525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6897.204184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6897.233482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.919732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.999976] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6908.393797] LustreError: 204392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.424238] LustreError: 204392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6913.522861] LustreError: 204393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.589278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.105043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6920.878405] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6928.001988] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6932.436769] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6940.599107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.300104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.373633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6948.734451] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6955.958726] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6965.474286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.529310] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6965.808471] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.683992] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6981.337410] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6983.608654] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6996.005176] Lustre: Permanently deactivating lustre-OST0001 [ 6996.038188] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7005.848727] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7016.317818] Lustre: Permanently reactivating lustre-OST0001 [ 7016.359871] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7016.371863] Lustre: Skipped 2 previous similar messages [ 7024.735285] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.748099] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7026.824969] Lustre: server umount lustre-OST0001 complete [ 7030.250153] LustreError: 205991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7030.758108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7030.774235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.803712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.777234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.783459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.804837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7035.362259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7035.363418] LustreError: 205991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.384254] Lustre: Skipped 1 previous similar message [ 7036.683919] Lustre: server umount lustre-OST0000 complete [ 7040.481614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.482755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7040.502845] Lustre: Skipped 1 previous similar message [ 7045.089832] LustreError: 207880:0:(osp_dev.c:693:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 7050.722790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.730935] Lustre: Skipped 4 previous similar messages [ 7050.788553] Lustre: server umount lustre-MDT0000 complete [ 7054.194634] LustreError: 204374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713071 with bad export cookie 7944336905902921407 [ 7054.207543] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7054.209140] LustreError: 204374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7054.585623] Lustre: server umount lustre-MDT0001 complete [ 7069.865946] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7073.635958] Key type lgssc unregistered [ 7073.987716] LNet: 209019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7073.993990] LNetError: 209019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7074.009385] LNet: Removed LNI 192.168.204.148@tcp [ 7074.939562] Key type .llcrypt unregistered [ 7074.947859] Key type ._llcrypt unregistered [ 7120.785826] Key type ._llcrypt registered [ 7120.788782] Key type .llcrypt registered [ 7120.878348] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7121.883085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.125264] alg: No test for adler32 (adler32-zlib) [ 7123.308952] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7123.616587] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7125.351211] Key type lgssc registered [ 7126.506615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7138.131412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7138.171988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.928747] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7139.972582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7140.252151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.890434] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7157.808195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7157.938350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7157.988166] Lustre: 211210:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7158.281476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7163.356096] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7170.198223] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7175.947449] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7184.096614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.396934] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7184.630498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.648767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7190.947717] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7197.956396] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7202.405896] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7205.853859] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7214.137078] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7222.338308] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7231.187349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7251.681777] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7251.696583] Lustre: Skipped 2 previous similar messages [ 7288.781985] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7288.792164] Lustre: Skipped 4 previous similar messages [ 7353.507221] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7353.515333] Lustre: Skipped 7 previous similar messages [ 7369.698790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7369.704293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.722377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.213541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.216018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7370.233452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.246597] Lustre: Skipped 1 previous similar message [ 7375.336529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7375.345224] Lustre: Skipped 1 previous similar message [ 7375.995880] Lustre: server umount lustre-OST0000 complete [ 7380.453038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.458996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7380.470459] Lustre: Skipped 1 previous similar message [ 7380.491957] Lustre: Skipped 1 previous similar message [ 7385.572186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.586843] Lustre: Skipped 1 previous similar message [ 7385.804706] Lustre: server umount lustre-MDT0000 complete [ 7389.302609] LustreError: 211203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713406 with bad export cookie 5973165504023435235 [ 7389.306181] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7389.311933] LustreError: 211203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7389.712989] Lustre: server umount lustre-MDT0001 complete [ 7402.084905] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7404.822616] Key type lgssc unregistered [ 7405.264484] LNet: 215653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.274668] LNetError: 215653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7405.321359] LNet: Removed LNI 192.168.204.148@tcp [ 7406.451731] Key type .llcrypt unregistered [ 7406.465757] Key type ._llcrypt unregistered [ 7435.814960] Key type ._llcrypt registered [ 7435.818510] Key type .llcrypt registered [ 7435.994722] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7437.286498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7437.607439] alg: No test for adler32 (adler32-zlib) [ 7438.858637] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7439.047755] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7440.711212] Key type lgssc registered [ 7442.680856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7450.963424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7450.986282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.492204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7457.913205] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7462.880534] LustreError: 217051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7462.897894] LustreError: 217051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7468.009408] LustreError: 217052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7468.207420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.698453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.501394] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7481.362859] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7487.308799] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7495.769954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.257661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.289446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7504.389420] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7511.428812] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7533.036231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7533.051562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.077923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.564088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7534.568899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.584917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.615586] Lustre: Skipped 1 previous similar message [ 7539.686454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7539.701131] Lustre: Skipped 1 previous similar message [ 7544.287852] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7544.782919] Lustre: server umount lustre-OST0000 complete [ 7555.041158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.067997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7555.079358] Lustre: Skipped 1 previous similar message [ 7557.185354] Lustre: server umount lustre-MDT0000 complete [ 7560.163116] LustreError: 217052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.196860] LustreError: 217052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7562.111182] LustreError: 217031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779713579 with bad export cookie 2231569083350331454 [ 7562.114414] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7562.126684] LustreError: 217031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7562.942669] Lustre: server umount lustre-MDT0001 complete [ 7574.347741] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7577.593509] Key type lgssc unregistered [ 7578.015725] LNet: 220387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7578.027757] LNetError: 220387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7578.049418] LNet: Removed LNI 192.168.204.148@tcp [ 7579.264205] Key type .llcrypt unregistered [ 7579.266954] Key type ._llcrypt unregistered [ 7604.341788] Key type ._llcrypt registered [ 7604.345345] Key type .llcrypt registered [ 7604.426271] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7605.191343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7605.259664] alg: No test for adler32 (adler32-zlib) [ 7606.497650] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7606.753994] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7608.495242] Key type lgssc registered [ 7609.687721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7882.434826] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7885.682380] Key type lgssc unregistered [ 7886.020071] LNet: 222382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7886.032963] LNetError: 222382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7886.051041] LNet: Removed LNI 192.168.204.148@tcp [ 7886.826614] Key type .llcrypt unregistered [ 7886.829572] Key type ._llcrypt unregistered [ 7917.233778] Key type ._llcrypt registered [ 7917.236597] Key type .llcrypt registered [ 7917.395506] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7934.319521] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7935.233387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7935.399787] alg: No test for adler32 (adler32-zlib) [ 7936.474329] Lustre: Lustre: Build Version: 2.17.53_24_gdcce612 [ 7936.729264] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7938.423434] Key type lgssc registered [ 7939.572982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7974.715685] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7674 sec ========== 08:59:50 (1779713990) [ 7976.808970] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:59:52 (1779713992) === [ 7981.495925] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:59:56 (1779713996) === [ 8026.879686] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 8030.092249] Key type lgssc unregistered [ 8030.404889] LNet: 227573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8030.412091] LNetError: 227573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8030.450180] LNet: Removed LNI 192.168.204.148@tcp [ 8031.418586] Key type .llcrypt unregistered [ 8031.422715] Key type ._llcrypt unregistered