[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514489077 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001022] APIC: Switch to symmetric I/O mode setup [ 0.003229] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008490] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011143] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.014025] SELinux: Initializing. [ 0.015083] *** VALIDATE selinux *** [ 0.023236] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027769] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029044] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032067] *** VALIDATE tmpfs *** [ 0.034113] *** VALIDATE proc *** [ 0.035236] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044166] debug: unmapping init [mem 0xffffffffbbe59000-0xffffffffbbe60fff] [ 0.047173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048692] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049037] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053011] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056306] rcu: Hierarchical SRCU implementation. [ 0.058466] smp: Bringing up secondary CPUs ... [ 0.059601] x86: Booting SMP configuration: [ 0.060028] .... node #0, CPUs: #1 #2 #3 [ 0.065285] smp: Brought up 1 node, 4 CPUs [ 0.067020] smpboot: Max logical packages: 1 [ 0.068015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230040] node 0 deferred pages initialised in 159ms [ 0.233171] devtmpfs: initialized [ 0.234338] x86/mm: Memory block size: 128MB [ 0.236546] gcov: version magic: 0x41383552 [ 0.241198] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244296] pinctrl core: initialized pinctrl subsystem [ 0.247238] [ 0.247883] ************************************************************* [ 0.250015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253015] ** ** [ 0.255011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257013] ** ** [ 0.259012] ** This means that this kernel is built to expose internal ** [ 0.261016] ** IOMMU data structures, which may compromise security on ** [ 0.263016] ** your system. ** [ 0.265012] ** ** [ 0.267013] ** If you see this message and you are not debugging the ** [ 0.268011] ** kernel, report this immediately to your vendor! ** [ 0.270014] ** ** [ 0.272017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274013] ************************************************************* [ 0.276700] NET: Registered protocol family 16 [ 0.279116] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.281107] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.284282] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288131] cpuidle: using governor menu [ 0.290845] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.293523] PCI: Using configuration type 1 for base access [ 0.295125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.313032] cryptd: max_cpu_qlen set to 1000 [ 0.316369] ACPI: Added _OSI(Module Device) [ 0.318031] ACPI: Added _OSI(Processor Device) [ 0.319017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.321022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326772] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.337412] ACPI: Interpreter enabled [ 0.339057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.341014] ACPI: Using IOAPIC for interrupt routing [ 0.344111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.348384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.360809] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.364094] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.367028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.372127] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.381508] acpiphp: Slot [2] registered [ 0.383438] acpiphp: Slot [5] registered [ 0.387267] acpiphp: Slot [6] registered [ 0.388126] acpiphp: Slot [7] registered [ 0.390150] acpiphp: Slot [8] registered [ 0.392181] acpiphp: Slot [9] registered [ 0.395165] acpiphp: Slot [10] registered [ 0.397142] acpiphp: Slot [3] registered [ 0.399122] acpiphp: Slot [4] registered [ 0.401122] acpiphp: Slot [11] registered [ 0.403123] acpiphp: Slot [12] registered [ 0.405211] acpiphp: Slot [13] registered [ 0.406104] acpiphp: Slot [14] registered [ 0.408103] acpiphp: Slot [15] registered [ 0.409111] acpiphp: Slot [16] registered [ 0.411113] acpiphp: Slot [17] registered [ 0.412101] acpiphp: Slot [18] registered [ 0.414100] acpiphp: Slot [19] registered [ 0.415123] acpiphp: Slot [20] registered [ 0.417106] acpiphp: Slot [21] registered [ 0.419125] acpiphp: Slot [22] registered [ 0.420108] acpiphp: Slot [23] registered [ 0.422177] acpiphp: Slot [24] registered [ 0.424174] acpiphp: Slot [25] registered [ 0.425144] acpiphp: Slot [26] registered [ 0.427154] acpiphp: Slot [27] registered [ 0.429158] acpiphp: Slot [28] registered [ 0.431173] acpiphp: Slot [29] registered [ 0.433149] acpiphp: Slot [30] registered [ 0.435198] acpiphp: Slot [31] registered [ 0.437278] PCI host bridge to bus 0000:00 [ 0.438033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.441037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.447039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.450053] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.454030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.456181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460209] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.463432] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.473881] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.479016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.482026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.484012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.486020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.489380] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.493743] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.496054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.500000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.506028] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.520023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.525024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.532477] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.543023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.550021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.565019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.576324] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.583092] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.592020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.614023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.631262] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.636017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.664018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.674499] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.684016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.692017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.722020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.737821] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.746025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.759024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.781023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.799248] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.810016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.819017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.835019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.848534] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.850429] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.853533] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.856504] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.859350] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.864139] iommu: Default domain type: Passthrough [ 0.869476] SCSI subsystem initialized [ 0.871176] ACPI: bus type USB registered [ 0.872103] usbcore: registered new interface driver usbfs [ 0.874108] usbcore: registered new interface driver hub [ 0.876087] usbcore: registered new device driver usb [ 0.878169] pps_core: LinuxPPS API ver. 1 registered [ 0.880017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.884071] PTP clock support registered [ 0.886124] EDAC MC: Ver: 3.0.0 [ 0.888198] PCI: Using ACPI for IRQ routing [ 0.891155] NetLabel: Initializing [ 0.892017] NetLabel: domain hash size = 128 [ 0.893026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.895128] NetLabel: unlabeled traffic allowed by default [ 0.898114] vgaarb: loaded [ 0.899711] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.902018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.908005] clocksource: Switched to clocksource kvm-clock [ 1.017771] VFS: Disk quotas dquot_6.6.0 [ 1.019548] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.021939] *** VALIDATE ramfs *** [ 1.023238] *** VALIDATE hugetlbfs *** [ 1.024752] pnp: PnP ACPI init [ 1.027770] pnp: PnP ACPI: found 6 devices [ 1.045438] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.048796] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.050760] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.052638] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.055181] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.057552] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.060281] NET: Registered protocol family 2 [ 1.062880] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.067881] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.071813] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.077632] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.082152] TCP: Hash tables configured (established 65536 bind 65536) [ 1.085323] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.088816] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.092081] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.096245] NET: Registered protocol family 1 [ 1.099327] RPC: Registered named UNIX socket transport module. [ 1.101742] RPC: Registered udp transport module. [ 1.103572] RPC: Registered tcp transport module. [ 1.105444] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.108179] NET: Registered protocol family 44 [ 1.109756] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.112056] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.114507] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.117248] PCI: CLS 0 bytes, default 64 [ 1.119075] Unpacking initramfs... [ 2.637203] debug: unmapping init [mem 0xffff9d183cc54000-0xffff9d183ffbffff] [ 2.641755] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.644429] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.647727] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.164750] Initialise system trusted keyrings [ 3.166832] Key type blacklist registered [ 3.169824] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.181790] zbud: loaded [ 3.185349] *** VALIDATE nfs *** [ 3.186776] *** VALIDATE nfs4 *** [ 3.188863] pstore: using deflate compression [ 3.193651] Platform Keyring initialized [ 3.329642] NET: Registered protocol family 38 [ 3.334141] Key type asymmetric registered [ 3.335887] Asymmetric key parser 'x509' registered [ 3.338199] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.341963] io scheduler mq-deadline registered [ 3.344261] io scheduler kyber registered [ 3.345840] io scheduler bfq registered [ 3.348681] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.352249] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.357250] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.359827] ACPI: Power Button [PWRF] [ 3.365096] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.372168] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.388776] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.404915] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.429241] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.461742] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.493879] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.499385] Non-volatile memory driver v1.3 [ 3.501164] Linux agpgart interface v0.103 [ 3.543274] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.546122] vda: detected capacity change from 0 to 74760192 [ 3.561731] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.564382] vdb: detected capacity change from 0 to 1073741824 [ 3.578878] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.581488] vdc: detected capacity change from 0 to 2621440000 [ 3.597224] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.600251] vdd: detected capacity change from 0 to 2621440000 [ 3.619139] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.622303] vde: detected capacity change from 0 to 4294967296 [ 3.639735] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.643538] vdf: detected capacity change from 0 to 4294967296 [ 3.652285] libphy: Fixed MDIO Bus: probed [ 3.657980] usbcore: registered new interface driver usbserial_generic [ 3.660404] usbserial: USB Serial support registered for generic [ 3.663326] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.668440] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.671488] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.675882] mousedev: PS/2 mouse device common for all mice [ 3.680468] rtc_cmos 00:05: RTC can wake from S4 [ 3.683822] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.686232] rtc_cmos 00:05: registered as rtc0 [ 3.691048] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.694704] intel_pstate: CPU model not supported [ 3.697868] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.700607] hid: raw HID events driver (C) Jiri Kosina [ 3.706156] usbcore: registered new interface driver usbhid [ 3.709269] usbhid: USB HID core driver [ 3.711527] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.711632] drop_monitor: Initializing network drop monitor service [ 3.718605] Initializing XFRM netlink socket [ 3.720980] NET: Registered protocol family 10 [ 3.726658] Segment Routing with IPv6 [ 3.728570] NET: Registered protocol family 17 [ 3.730535] mpls_gso: MPLS GSO support [ 3.738107] RAS: Correctable Errors collector initialized. [ 3.740307] AVX version of gcm_enc/dec engaged. [ 3.742024] AES CTR mode by8 optimization enabled [ 3.837551] sched_clock: Marking stable (3837518349, 0)->(4767125622, -929607273) [ 3.841644] registered taskstats version 1 [ 3.843538] Loading compiled-in X.509 certificates [ 3.845619] zswap: loaded using pool lzo/zbud [ 3.871224] Key type big_key registered [ 3.884910] Key type encrypted registered [ 3.886737] ima: No TPM chip found, activating TPM-bypass! [ 3.889133] ima: Allocated hash algorithm: sha1 [ 3.891113] ima: No architecture policies found [ 3.893446] evm: Initialising EVM extended attributes: [ 3.895414] evm: security.selinux [ 3.898075] evm: security.ima [ 3.899378] evm: security.capability [ 3.900882] evm: HMAC attrs: 0x1 [ 3.905075] rtc_cmos 00:05: setting system clock to 2026-08-29 09:08:10 UTC (1787994490) [ 3.911627] debug: unmapping init [mem 0xffffffffbce03000-0xffffffffbcffffff] [ 3.915103] debug: unmapping init [mem 0xffffffffbbb82000-0xffffffffbbe58fff] [ 3.924110] Write protecting the kernel read-only data: 28672k [ 3.928711] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 3.932317] debug: unmapping init [mem 0xffffffffbab14000-0xffffffffbabfffff] [ 3.972853] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.984434] systemd[1]: Detected virtualization kvm. [ 3.986767] systemd[1]: Detected architecture x86-64. [ 3.988932] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.016619] systemd[1]: No hostname configured. [ 4.018484] systemd[1]: Set hostname to . [ 4.020673] random: systemd: uninitialized urandom read (16 bytes read) [ 4.023396] systemd[1]: Initializing machine ID from random generator. [ 4.186697] random: systemd: uninitialized urandom read (16 bytes read) [ 4.188893] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.193379] random: systemd: uninitialized urandom read (16 bytes read) [ 4.196240] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.200868] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.904510] device-mapper: uevent: version 1.0.3 [ 4.907136] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.738504] virtio_net virtio0 ens2: renamed from eth0 [ 5.747543] random: fast init done [ 5.829527] scsi host0: ata_piix [ 5.870455] scsi host1: ata_piix [ 5.873150] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.877057] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.714423] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.410325] random: crng init done [ 10.414967] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.075640] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.391097] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.680633] SELinux: Disabled at runtime. [ 12.743514] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.753634] systemd[1]: Detected virtualization kvm. [ 12.755740] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.246489] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.249607] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.256509] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.262413] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.267626] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.275601] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.283427] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 13.492817] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.786163] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.218240] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.243776] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.396350] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.410346] EDAC sbridge: Ver: 1.1.2 [ 16.095850] Key type dns_resolver registered [ 16.407490] NFS: Registering the id_resolver key type [ 16.410088] Key type id_resolver registered [ 16.412056] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 60.453575] hrtimer: interrupt took 6381362 ns [ 69.103491] libcfs: loading out-of-tree module taints kernel. [ 69.147157] Key type ._llcrypt registered [ 69.149853] Key type .llcrypt registered [ 69.265551] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 88.901588] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 90.308223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.322593] alg: No test for adler32 (adler32-zlib) [ 91.971250] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 92.776691] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 94.584593] Key type lgssc registered [ 96.380328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.726402] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.397685] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 174.165909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 174.223784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.490114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 175.520671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 175.615795] Lustre: lustre-MDT0000: new disk, initializing [ 175.697755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.724725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.195884] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.137674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.237471] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.272299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 196.278180] Lustre: Skipped 1 previous similar message [ 196.423907] Lustre: lustre-MDT0001: new disk, initializing [ 196.529917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.568443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.581340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 201.397902] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.462886] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 216.596464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.044070] Lustre: lustre-OST0000: new disk, initializing [ 217.049024] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.061023] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.147519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.603347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.615176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.704571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 223.802653] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.151175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.301757] Lustre: lustre-OST0001: new disk, initializing [ 239.307479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.314751] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.414663] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.781724] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.493117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.503571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.595291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.118388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.467526] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.073542] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.933046] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 286.963823] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 289.926718] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 292.445931] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 294.208867] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 29 05:12:58 EDT 2026 [ 313.217208] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 314.545281] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.720968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.724108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.736784] Lustre: Skipped 2 previous similar messages [ 330.748191] Lustre: Skipped 3 previous similar messages [ 335.842013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.847141] Lustre: Skipped 3 previous similar messages [ 336.272562] Lustre: server umount lustre-MDT0000 complete [ 340.403650] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994826 with bad export cookie 17442463357062978431 [ 340.405613] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.418350] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.961497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.978307] Lustre: Skipped 1 previous similar message [ 340.984137] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.993080] Lustre: Skipped 1 previous similar message [ 346.080238] LustreError: 8041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.080361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.107143] LustreError: 8041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 346.125484] Lustre: Skipped 1 previous similar message [ 346.659522] Lustre: server umount lustre-MDT0001 complete [ 351.602994] Lustre: server umount lustre-OST0000 complete [ 356.136267] Lustre: server umount lustre-OST0001 complete [ 365.074211] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 376.431829] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 413.263347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.614869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.663223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.779579] Lustre: lustre-MDT0000: new disk, initializing [ 413.885914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.909454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.541412] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 428.908152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.043189] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 429.137681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 429.143812] Lustre: Skipped 1 previous similar message [ 429.287219] Lustre: lustre-MDT0001: new disk, initializing [ 429.423370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.472067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 429.485296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.104944] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 444.124532] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 449.323867] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 455.314636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.561157] Lustre: lustre-OST0000: new disk, initializing [ 455.564885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.570737] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.660113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.808739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.832232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.975168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.455958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 471.972181] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 477.256173] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.506593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 481.788717] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 481.984565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 487.904916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.912570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.912583] Lustre: Skipped 1 previous similar message [ 487.914105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.927131] LustreError: Skipped 1 previous similar message [ 487.941312] Lustre: Skipped 1 previous similar message [ 498.146533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.147983] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 498.163166] Lustre: Skipped 3 previous similar messages [ 498.438551] Lustre: server umount lustre-OST0000 complete [ 503.267856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.278340] Lustre: Skipped 1 previous similar message [ 508.335821] Lustre: server umount lustre-MDT0000 complete [ 508.385104] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.395666] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 511.762745] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994998 with bad export cookie 17442463357062979964 [ 511.777097] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.780190] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 512.120510] Lustre: server umount lustre-MDT0001 complete [ 515.572424] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:16:40 (1787995000) [ 521.794590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.657527] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.788542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.854951] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.925799] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 532.965642] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.465713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.274158] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 551.459301] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 558.307751] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.081719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.434944] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 580.811291] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 592.887492] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 592.899630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.919908] Lustre: Skipped 1 previous similar message [ 592.942802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.956933] Lustre: Skipped 2 previous similar messages [ 598.228420] Lustre: server umount lustre-OST0000 complete [ 603.386502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.399116] Lustre: Skipped 2 previous similar messages [ 609.676399] Lustre: server umount lustre-MDT0000 complete [ 613.532803] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995100 with bad export cookie 17442463357062980846 [ 613.536429] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.541708] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 613.759546] Lustre: server umount lustre-MDT0001 complete [ 625.632996] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 628.848973] Key type lgssc unregistered [ 629.257772] LNet: 22845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.267791] LNetError: 22845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.280634] LNet: Removed LNI 192.168.204.154@tcp [ 630.404154] Key type .llcrypt unregistered [ 630.406291] Key type ._llcrypt unregistered [ 654.144335] Key type ._llcrypt registered [ 654.148534] Key type .llcrypt registered [ 654.293365] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 655.098803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.182241] alg: No test for adler32 (adler32-zlib) [ 656.369198] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 656.684152] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 658.399255] Key type lgssc registered [ 659.520652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.069821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 668.128785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.791502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.326832] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 679.918874] LustreError: 24262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.945265] LustreError: 24262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 684.087130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.679720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 689.107888] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 696.786169] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 702.989238] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 710.150556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 710.691450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.822822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 717.577189] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 724.989987] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 745.444568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.454765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 745.462866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 745.465422] LustreError: Skipped 1 previous similar message [ 745.475818] Lustre: Skipped 1 previous similar message [ 749.397884] Lustre: server umount lustre-OST0000 complete [ 755.680881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.692981] Lustre: Skipped 1 previous similar message [ 755.698979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.709536] Lustre: Skipped 1 previous similar message [ 759.993767] Lustre: server umount lustre-MDT0000 complete [ 760.807427] LustreError: 24258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.831509] LustreError: 24258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 764.029882] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995250 with bad export cookie 10890460646539230041 [ 764.047579] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 764.474942] Lustre: server umount lustre-MDT0001 complete [ 776.813518] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 780.264729] Key type lgssc unregistered [ 780.578754] LNet: 28240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.584830] LNetError: 28240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.625521] LNet: Removed LNI 192.168.204.154@tcp [ 781.778873] Key type .llcrypt unregistered [ 781.785705] Key type ._llcrypt unregistered [ 808.262655] Key type ._llcrypt registered [ 808.265252] Key type .llcrypt registered [ 808.378363] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 809.995405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 810.022407] alg: No test for adler32 (adler32-zlib) [ 811.253392] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 811.627546] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 813.351357] Key type lgssc registered [ 814.761419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 824.678761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 824.702976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.268484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.674375] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 835.563706] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.594286] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 840.678976] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.471166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.962772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 848.345386] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 858.315778] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 865.481608] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 881.439508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.026625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.118112] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 897.403870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 900.070056] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 912.865391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 912.887556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.916954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 915.132303] Lustre: server umount lustre-OST0000 complete [ 923.108233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.111839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.133460] Lustre: Skipped 2 previous similar messages [ 923.158339] Lustre: Skipped 2 previous similar messages [ 926.256828] Lustre: server umount lustre-MDT0000 complete [ 928.229645] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.243242] LustreError: 29655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 930.962419] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995417 with bad export cookie 8271109301267338951 [ 930.973607] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.981208] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 931.359778] Lustre: server umount lustre-MDT0001 complete [ 945.143139] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 948.546616] Key type lgssc unregistered [ 948.963053] LNet: 33539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.979746] LNetError: 33539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.015390] LNet: Removed LNI 192.168.204.154@tcp [ 950.096829] Key type .llcrypt unregistered [ 950.098510] Key type ._llcrypt unregistered [ 975.460189] Key type ._llcrypt registered [ 975.463525] Key type .llcrypt registered [ 975.623881] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 976.754537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 976.919837] alg: No test for adler32 (adler32-zlib) [ 977.964723] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 978.297559] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 980.023190] Key type lgssc registered [ 981.450182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.430471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.452140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.027961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.418082] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1001.451234] LustreError: 34956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.482785] LustreError: 34956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1006.560987] LustreError: 34957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.638242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.079716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.221486] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1025.819985] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1035.460026] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1043.124495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.436187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.232434] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1059.904532] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1061.907211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1072.096866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.105726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1072.115329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1072.121931] Lustre: Skipped 1 previous similar message [ 1072.165700] Lustre: Skipped 1 previous similar message [ 1076.890385] Lustre: server umount lustre-OST0000 complete [ 1082.343145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.353892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.359290] Lustre: Skipped 1 previous similar message [ 1082.374636] Lustre: Skipped 1 previous similar message [ 1087.450465] Lustre: server umount lustre-MDT0000 complete [ 1087.458760] LustreError: 34957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.554256] LustreError: 34956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.576732] LustreError: 34956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1091.854110] LustreError: 34936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995578 with bad export cookie 9541377980750285565 [ 1091.876940] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.240390] Lustre: server umount lustre-MDT0001 complete [ 1103.973336] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1106.568500] Key type lgssc unregistered [ 1106.862666] LNet: 38495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.875050] LNetError: 38495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.897332] LNet: Removed LNI 192.168.204.154@tcp [ 1107.734140] Key type .llcrypt unregistered [ 1107.735947] Key type ._llcrypt unregistered [ 1130.485569] Key type ._llcrypt registered [ 1130.491695] Key type .llcrypt registered [ 1130.657541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1131.763374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1131.932989] alg: No test for adler32 (adler32-zlib) [ 1133.049094] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1133.253266] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1134.911229] Key type lgssc registered [ 1136.227038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.105608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.147062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.602827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1150.140227] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1156.064203] LustreError: 39903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.081364] LustreError: 39903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1157.091021] LustreError: 39904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.862292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.256264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1166.955588] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1175.712222] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1182.117892] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1189.248864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1189.558520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.615134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1195.693591] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1203.993598] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1216.800434] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1217.528228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.532810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1217.537618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1217.548848] Lustre: Skipped 1 previous similar message [ 1217.578544] LustreError: Skipped 1 previous similar message [ 1220.420455] Lustre: server umount lustre-OST0000 complete [ 1243.105675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.127700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1243.139312] Lustre: Skipped 2 previous similar messages [ 1248.052516] Lustre: server umount lustre-MDT0000 complete [ 1248.228809] LustreError: 40632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.261312] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995739 with bad export cookie 10619624186766719270 [ 1253.270477] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.282625] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1259.734874] Lustre: server umount lustre-MDT0001 complete [ 1272.269978] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1275.331500] Key type lgssc unregistered [ 1275.664241] LNet: 43615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.681251] LNetError: 43615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.702108] LNet: Removed LNI 192.168.204.154@tcp [ 1276.540072] Key type .llcrypt unregistered [ 1276.546911] Key type ._llcrypt unregistered [ 1302.660459] Key type ._llcrypt registered [ 1302.662558] Key type .llcrypt registered [ 1302.789113] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1303.933153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1303.972167] alg: No test for adler32 (adler32-zlib) [ 1305.117798] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1305.356375] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1307.103230] Key type lgssc registered [ 1308.295896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.932931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.958661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.481429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.483582] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1328.613118] LustreError: 45032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.638644] LustreError: 45032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1331.782165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.696255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1337.733895] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1348.276152] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1355.400304] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1362.891940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1363.288978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.344903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1370.289995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1379.221143] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1385.761549] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1388.002459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.028921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1388.030599] Lustre: Skipped 1 previous similar message [ 1388.056117] Lustre: Skipped 2 previous similar messages [ 1390.852094] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 1391.907467] Lustre: server umount lustre-MDT0000 complete [ 1393.122023] LustreError: 45759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.150357] LustreError: 45759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1395.641824] LustreError: 45011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995882 with bad export cookie 1880273950444471039 [ 1395.651375] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1395.654176] LustreError: 45011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1395.968390] LustreError: 45027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.001334] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [ 1398.241468] LustreError: 45028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.245778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1398.276181] Lustre: Skipped 1 previous similar message [ 1400.292264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1400.298359] Lustre: Skipped 1 previous similar message [ 1402.206952] Lustre: server umount lustre-MDT0001 complete [ 1423.652853] Lustre: server umount lustre-OST0000 complete [ 1441.229376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1444.299492] Key type lgssc unregistered [ 1444.532673] LNet: 48895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.541245] LNetError: 48895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.561287] LNet: Removed LNI 192.168.204.154@tcp [ 1445.485707] Key type .llcrypt unregistered [ 1445.487926] Key type ._llcrypt unregistered [ 1474.821456] Key type ._llcrypt registered [ 1474.823265] Key type .llcrypt registered [ 1475.029812] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1476.654576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.849064] alg: No test for adler32 (adler32-zlib) [ 1478.095433] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1478.499686] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1480.279181] Key type lgssc registered [ 1481.617723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1490.979627] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1491.002844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.591583] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1511.711326] LustreError: 50283:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 1538.527724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.474413] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1557.057188] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1725.678773] Lustre: server umount lustre-OST0000 complete [ 1746.837768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1750.141497] Key type lgssc unregistered [ 1750.490677] LNet: 51964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.495136] LNetError: 51964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.516170] LNet: Removed LNI 192.168.204.154@tcp [ 1751.735185] Key type .llcrypt unregistered [ 1751.742250] Key type ._llcrypt unregistered [ 1779.264582] Key type ._llcrypt registered [ 1779.271910] Key type .llcrypt registered [ 1779.436479] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1781.050704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1781.203793] alg: No test for adler32 (adler32-zlib) [ 1782.532966] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1783.076406] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1784.767946] Key type lgssc registered [ 1786.131170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.063365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.086484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.912921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.886113] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1807.331993] LustreError: 53378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.387401] LustreError: 53378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1812.448381] LustreError: 53379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.695688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.048214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1818.640164] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1827.873640] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1834.745498] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1843.499055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1844.032887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.069336] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1861.475708] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1867.236537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.238811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.414720] Lustre: server umount lustre-OST0000 complete [ 1877.484591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.503183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.516361] Lustre: Skipped 2 previous similar messages [ 1882.594924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.607230] Lustre: Skipped 1 previous similar message [ 1886.691297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1886.697381] Lustre: Skipped 1 previous similar message [ 1889.247143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1889.600745] Lustre: server umount lustre-MDT0000 complete [ 1891.809816] LustreError: 53378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.845154] LustreError: 53378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1894.209947] LustreError: 53359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996380 with bad export cookie 663488552040382785 [ 1894.226342] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1894.534708] Lustre: server umount lustre-MDT0001 complete [ 1907.173891] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1910.645287] Key type lgssc unregistered [ 1911.132696] LNet: 56915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.154940] LNetError: 56915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.169797] LNet: Removed LNI 192.168.204.154@tcp [ 1912.280553] Key type .llcrypt unregistered [ 1912.284048] Key type ._llcrypt unregistered [ 1938.408423] Key type ._llcrypt registered [ 1938.410773] Key type .llcrypt registered [ 1938.505651] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1939.860259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1939.959842] alg: No test for adler32 (adler32-zlib) [ 1941.310828] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1941.575705] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1943.255213] Key type lgssc registered [ 1944.940525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.765442] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.799796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1970.464065] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1975.584875] LustreError: 58302:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2002.335564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.172588] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2017.743337] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2025.867535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.879333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.057768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2032.099047] LustreError: 58328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.122076] LustreError: 58328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2032.726511] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2037.218769] LustreError: 58327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.254952] LustreError: 58327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2042.343790] LustreError: 59780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.360774] LustreError: 59780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2043.018610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.584084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.456827] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2057.095199] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.585315] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.945092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.960270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.988753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2069.475260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2069.477940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2069.484067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.599277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2074.603442] Lustre: Skipped 1 previous similar message [ 2079.721075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2079.730563] Lustre: Skipped 1 previous similar message [ 2080.223557] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2080.435343] Lustre: server umount lustre-OST0000 complete [ 2104.628162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.631786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.655731] Lustre: Skipped 1 previous similar message [ 2104.679093] Lustre: Skipped 2 previous similar messages [ 2111.039743] Lustre: server umount lustre-MDT0000 complete [ 2115.044462] LustreError: 61034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.066636] LustreError: 61034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2115.985216] LustreError: 58309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996602 with bad export cookie 1032589700365942411 [ 2115.997620] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.005322] LustreError: 58309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.723102] Lustre: server umount lustre-MDT0001 complete [ 2138.271805] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2142.056198] Key type lgssc unregistered [ 2142.595925] LNet: 62052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.609850] LNetError: 62052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.641463] LNet: Removed LNI 192.168.204.154@tcp [ 2143.735321] Key type .llcrypt unregistered [ 2143.741557] Key type ._llcrypt unregistered [ 2172.216555] Key type ._llcrypt registered [ 2172.218971] Key type .llcrypt registered [ 2172.316307] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2172.857760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.918454] alg: No test for adler32 (adler32-zlib) [ 2174.026166] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2174.297823] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2176.023520] Key type lgssc registered [ 2177.665474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2187.402601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2187.461529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.174751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2195.561887] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2198.503111] LustreError: 63480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.547180] LustreError: 63480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2203.618162] LustreError: 63479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.322805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.836893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2213.211161] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2223.292178] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2230.785519] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2239.511050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.926441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.538780] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2256.509099] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2259.393248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2278.881750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.899530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.920671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.905241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.909646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.909809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.915633] Lustre: Skipped 1 previous similar message [ 2285.037872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.143294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2290.164254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.173915] Lustre: Skipped 1 previous similar message [ 2290.440413] Lustre: server umount lustre-OST0000 complete [ 2299.879115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2299.899524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.926211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.860453] Lustre: server umount lustre-MDT0000 complete [ 2305.505622] LustreError: 63480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.524504] LustreError: 63480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2306.325837] LustreError: 63461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996792 with bad export cookie 6030069382395756235 [ 2306.339288] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.349444] LustreError: 63461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2306.969213] Lustre: server umount lustre-MDT0001 complete [ 2319.851815] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2323.067977] Key type lgssc unregistered [ 2323.392385] LNet: 67042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.406392] LNetError: 67042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.423905] LNet: Removed LNI 192.168.204.154@tcp [ 2324.488122] Key type .llcrypt unregistered [ 2324.494443] Key type ._llcrypt unregistered [ 2352.814169] Key type ._llcrypt registered [ 2352.822121] Key type .llcrypt registered [ 2352.994055] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2354.531288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.612910] alg: No test for adler32 (adler32-zlib) [ 2355.681696] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2355.947362] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2357.712312] Key type lgssc registered [ 2359.170123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2370.620242] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 05:47:35 (1787996855) [ 2376.859901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.900827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.446165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.032721] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2388.962305] LustreError: 68745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.973071] LustreError: 68745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2393.481384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.051799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2399.114503] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2409.163673] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.410784] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2422.371092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2422.641714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.678496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2428.822603] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2437.619317] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2444.723515] Lustre: Failing over lustre-MDT0000 [ 2445.055688] Lustre: server umount lustre-MDT0000 complete [ 2445.285718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2445.302342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.383766] LustreError: 68745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.376807] LustreError: 68746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.391654] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.392579] LustreError: 68746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2454.498642] LustreError: 68746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.525378] LustreError: 68746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2458.031789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.251881] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.488302] Lustre: *** cfs_fail_loc=135, val=0*** [ 2458.636995] LustreError: 71836:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2458.650593] LustreError: 71836:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -2 [ 2458.664266] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2458.664266] [ 2458.677177] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2458.703320] LustreError: 71823:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2458.712522] LustreError: 71823:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2458.725747] LustreError: 71823:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2458.738922] LustreError: 71823:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2458.781484] Lustre: server umount lustre-MDT0000 complete [ 2458.785174] LustreError: 71823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2459.641818] LustreError: 68745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.660153] LustreError: 68745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2466.702137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.871661] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.884209] LustreError: 72181:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.154@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000031251c6e) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2466.913912] LustreError: 68737:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2467.164979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.235861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2471.171443] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2472.444435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2472.495340] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2472.587332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2472.999942] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2482.658278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.665561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.676730] Lustre: Skipped 2 previous similar messages [ 2482.685107] Lustre: Skipped 1 previous similar message [ 2487.792650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.804185] Lustre: Skipped 1 previous similar message [ 2488.539478] Lustre: server umount lustre-OST0000 complete [ 2492.897625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.904254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.913646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.920195] Lustre: Skipped 1 previous similar message [ 2498.025184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.391612] Lustre: server umount lustre-MDT0000 complete [ 2503.139097] LustreError: 68740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.155121] LustreError: 68740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2503.486165] LustreError: 68726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996990 with bad export cookie 3617666533393156882 [ 2503.508684] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.170672] Lustre: server umount lustre-MDT0001 complete [ 2518.228393] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2522.475719] Key type lgssc unregistered [ 2522.855674] LNet: 73619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.869074] LNetError: 73619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.904643] LNet: Removed LNI 192.168.204.154@tcp [ 2524.188626] Key type .llcrypt unregistered [ 2524.193524] Key type ._llcrypt unregistered [ 2554.924892] Key type ._llcrypt registered [ 2554.930408] Key type .llcrypt registered [ 2555.155979] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2556.406352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2556.486300] alg: No test for adler32 (adler32-zlib) [ 2558.159686] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2558.607786] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2560.376182] Key type lgssc registered [ 2561.928251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2571.025711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2571.042337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.532877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.109989] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2583.010948] LustreError: 75034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.029197] LustreError: 75034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2587.838065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.176266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.963167] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2602.681429] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2609.424166] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2617.756032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.243654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2625.586351] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2635.506453] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2642.809131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2651.303854] Lustre: Failing over lustre-MDT0000 [ 2651.430635] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 2651.779429] Lustre: server umount lustre-MDT0000 complete [ 2653.156328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.163798] Lustre: Skipped 2 previous similar messages [ 2653.164159] LustreError: 75031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.179244] LustreError: 75031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2656.518150] LustreError: 77125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.327459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.424775] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.602057] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2661.642319] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.54@tcp (not set up) [ 2661.788687] LustreError: 78146:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2661.795235] LustreError: 78146:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -9 [ 2661.804127] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2661.804127] [ 2661.812756] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2661.832804] LustreError: 78134:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2661.843227] LustreError: 78134:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2661.858728] LustreError: 78134:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2661.866654] LustreError: 78134:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2661.897760] Lustre: server umount lustre-MDT0000 complete [ 2661.900215] LustreError: 78134:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2663.394854] LustreError: 75035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.416080] LustreError: 75035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2668.515815] LustreError: 75034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.543346] LustreError: 75034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2669.487313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.642728] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2669.956707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.027560] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2671.885315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2675.081401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2675.182814] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2675.281616] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2675.366575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2685.431443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.440667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2685.444482] Lustre: Skipped 1 previous similar message [ 2685.463233] Lustre: Skipped 1 previous similar message [ 2690.496426] Lustre: server umount lustre-OST0000 complete [ 2695.649262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.653240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2695.669330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.676119] Lustre: Skipped 1 previous similar message [ 2700.771285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2700.789565] Lustre: Skipped 1 previous similar message [ 2701.695364] Lustre: server umount lustre-MDT0000 complete [ 2705.888538] LustreError: 75035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.934206] LustreError: 75035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2706.469770] LustreError: 79178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997193 with bad export cookie 16624335441407038756 [ 2706.475051] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.494045] LustreError: 79178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2707.089735] Lustre: server umount lustre-MDT0001 complete [ 2718.639875] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2721.436976] Key type lgssc unregistered [ 2721.842900] LNet: 79932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.848725] LNetError: 79932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.864529] LNet: Removed LNI 192.168.204.154@tcp [ 2722.847935] Key type .llcrypt unregistered [ 2722.855427] Key type ._llcrypt unregistered [ 2737.166838] Key type ._llcrypt registered [ 2737.170019] Key type .llcrypt registered [ 2737.386843] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2737.770493] Key type .llcrypt unregistered [ 2737.775482] Key type ._llcrypt unregistered [ 2765.363379] Key type ._llcrypt registered [ 2765.369687] Key type .llcrypt registered [ 2765.558940] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2766.563992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2766.615801] alg: No test for adler32 (adler32-zlib) [ 2767.737527] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2767.912045] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2769.560770] Key type lgssc registered [ 2770.843186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2780.687928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2780.705601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.173906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.247271] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2792.421858] LustreError: 81839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.433480] LustreError: 81839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2796.671539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.171619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.768916] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2809.778201] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2816.505562] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2824.542471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.025257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.085605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2831.238973] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2839.284582] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2853.858827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2853.864376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.891967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.953182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2857.955190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.682809] Lustre: server umount lustre-OST0000 complete [ 2868.193360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.197851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.204092] Lustre: Skipped 1 previous similar message [ 2868.215178] Lustre: Skipped 2 previous similar messages [ 2870.212087] Lustre: server umount lustre-MDT0000 complete [ 2873.320922] LustreError: 81836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.338950] LustreError: 81836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2874.063270] LustreError: 81821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997360 with bad export cookie 16804638533479579260 [ 2874.072172] LustreError: 81821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2874.073363] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2874.392873] Lustre: server umount lustre-MDT0001 complete [ 2886.117884] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2888.934981] Key type lgssc unregistered [ 2889.240507] LNet: 85398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2889.247570] LNetError: 85398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2889.281216] LNet: Removed LNI 192.168.204.154@tcp [ 2889.920547] Key type .llcrypt unregistered [ 2889.923180] Key type ._llcrypt unregistered [ 2914.806376] Key type ._llcrypt registered [ 2914.812722] Key type .llcrypt registered [ 2914.942585] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2916.444392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2916.475680] alg: No test for adler32 (adler32-zlib) [ 2917.442877] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2917.720629] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2919.479153] Key type lgssc registered [ 2920.961986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2928.350788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2928.363722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.768408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.642389] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2939.872420] LustreError: 86812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.911687] LustreError: 86812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2943.599277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.929956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.371250] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2957.565370] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2964.242247] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2971.634055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.931634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.031802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2979.014559] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2986.443082] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.777401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.783445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2999.793676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.806602] Lustre: Skipped 1 previous similar message [ 3004.905214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.910981] Lustre: Skipped 1 previous similar message [ 3010.015322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3010.022500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.041614] Lustre: Skipped 1 previous similar message [ 3010.246687] Lustre: server umount lustre-OST0000 complete [ 3015.138313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.144505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.153752] Lustre: Skipped 2 previous similar messages [ 3015.165605] Lustre: Skipped 1 previous similar message [ 3020.261113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.264015] Lustre: Skipped 1 previous similar message [ 3020.537797] Lustre: server umount lustre-MDT0000 complete [ 3024.183039] LustreError: 86792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997510 with bad export cookie 17737832234103428820 [ 3024.202882] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.453087] Lustre: server umount lustre-MDT0001 complete [ 3037.646974] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3040.416105] Key type lgssc unregistered [ 3040.633312] LNet: 90352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3040.645132] LNetError: 90352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3040.656347] LNet: Removed LNI 192.168.204.154@tcp [ 3041.589576] Key type .llcrypt unregistered [ 3041.591513] Key type ._llcrypt unregistered [ 3071.705517] Key type ._llcrypt registered [ 3071.718333] Key type .llcrypt registered [ 3071.840667] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3073.623672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.679109] alg: No test for adler32 (adler32-zlib) [ 3074.853964] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 3075.134390] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3076.783212] Key type lgssc registered [ 3077.759165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3086.426980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3086.458819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.109093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.063765] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3097.570831] LustreError: 91769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.587665] LustreError: 91769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3100.920653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.171677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3106.208130] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3113.993526] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3119.762051] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3127.366424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.724798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.858309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3135.616817] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3145.311529] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3162.596624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.607994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.616550] Lustre: Skipped 1 previous similar message [ 3162.620192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3162.621702] Lustre: Skipped 1 previous similar message [ 3167.724378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.757092] Lustre: Skipped 1 previous similar message [ 3168.006984] Lustre: server umount lustre-OST0000 complete [ 3172.843178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3172.851782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.870596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3172.883439] Lustre: Skipped 1 previous similar message [ 3177.952651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.969655] Lustre: Skipped 2 previous similar messages [ 3178.914682] Lustre: server umount lustre-MDT0000 complete [ 3183.076791] LustreError: 91769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.124793] LustreError: 91769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3183.539660] LustreError: 91749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997670 with bad export cookie 7568227597336550683 [ 3183.544032] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3183.554950] LustreError: 91749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3183.929259] Lustre: server umount lustre-MDT0001 complete [ 3197.550507] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3201.284697] Key type lgssc unregistered [ 3201.611372] LNet: 95342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3201.617297] LNetError: 95342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3201.645874] LNet: Removed LNI 192.168.204.154@tcp [ 3202.679186] Key type .llcrypt unregistered [ 3202.688440] Key type ._llcrypt unregistered [ 3226.865791] Key type ._llcrypt registered [ 3226.868521] Key type .llcrypt registered [ 3226.972619] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3228.274543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3228.350422] alg: No test for adler32 (adler32-zlib) [ 3229.338755] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 3229.622587] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3231.287144] Key type lgssc registered [ 3232.383533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3240.883897] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3240.908408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3256.547845] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3261.663271] LustreError: 96727:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 3288.287243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.496409] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3303.018502] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3309.158600] Lustre: server umount lustre-OST0000 complete [ 3318.961673] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:03:23 (1787997803) [ 3326.197259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.596166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.439534] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3332.064225] LustreError: 98367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.082873] LustreError: 98367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3337.199895] LustreError: 98366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3341.683583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.516571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3348.641421] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3357.881659] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3365.670989] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3374.367700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3374.992299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.113530] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3395.359516] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.481864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3411.936160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3411.947321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.963925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.978153] Lustre: Skipped 1 previous similar message [ 3411.992166] Lustre: Skipped 1 previous similar message [ 3414.862792] Lustre: server umount lustre-OST0000 complete [ 3422.178499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.194162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3422.209341] Lustre: Skipped 1 previous similar message [ 3425.742438] Lustre: server umount lustre-MDT0000 complete [ 3427.297252] LustreError: 100810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.334685] LustreError: 100810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3430.693621] LustreError: 98346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997917 with bad export cookie 9889755020931637921 [ 3430.704397] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.707863] LustreError: 98346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3431.142738] Lustre: server umount lustre-MDT0001 complete [ 3443.213339] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3446.442504] Key type lgssc unregistered [ 3446.776050] LNet: 102064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3446.787923] LNetError: 102064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3447.867969] LNet: Removed LNI 192.168.204.154@tcp [ 3448.721242] Key type .llcrypt unregistered [ 3448.723282] Key type ._llcrypt unregistered [ 3491.592778] Key type ._llcrypt registered [ 3491.600581] Key type .llcrypt registered [ 3491.759308] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3492.753193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3492.910410] alg: No test for adler32 (adler32-zlib) [ 3494.039504] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 3494.378843] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3496.167189] Key type lgssc registered [ 3497.297845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3505.048442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3505.074410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.653224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.007815] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3516.896322] LustreError: 104233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.917891] LustreError: 104233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3519.325780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.715923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3523.104460] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3531.351104] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3537.536876] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3545.098871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3545.567023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.745214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3552.636281] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3561.019991] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3574.760794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3574.772134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.810524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.288301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3575.297671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.731665] Lustre: server umount lustre-OST0000 complete [ 3585.514037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.527084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3585.547650] Lustre: Skipped 1 previous similar message [ 3585.557271] Lustre: Skipped 3 previous similar messages [ 3588.547863] Lustre: server umount lustre-MDT0000 complete [ 3590.633657] LustreError: 104962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.668040] LustreError: 104962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3592.739935] LustreError: 104972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998079 with bad export cookie 9028333976111918922 [ 3592.759632] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.767069] LustreError: 104972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3593.266441] Lustre: server umount lustre-MDT0001 complete [ 3606.121716] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3609.163070] Key type lgssc unregistered [ 3609.432489] LNet: 107771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3609.444550] LNetError: 107771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3609.453854] LNet: Removed LNI 192.168.204.154@tcp [ 3610.321564] Key type .llcrypt unregistered [ 3610.325382] Key type ._llcrypt unregistered [ 3630.959030] Key type ._llcrypt registered [ 3630.962458] Key type .llcrypt registered [ 3631.074727] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3632.459841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3632.492525] alg: No test for adler32 (adler32-zlib) [ 3633.659646] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 3633.952635] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3635.695254] Key type lgssc registered [ 3637.007739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3645.081951] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3645.098923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3660.705718] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3665.823950] LustreError: 109015:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 3692.575220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.782829] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3708.695822] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3716.387432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.736835] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3716.745614] LustreError: 110039:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3716.754692] LustreError: 110039:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3716.763974] LustreError: 110039:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3716.830888] Lustre: server umount lustre-MDT0000 complete [ 3716.833800] LustreError: 110039:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3735.519240] Lustre: 108353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787998206/real 1787998206] req@000000007f81571f x1874851916680576/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787998222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3735.562305] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3735.590508] LustreError: 109027:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3735.829898] Lustre: server umount lustre-OST0000 complete [ 3746.683615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 3756.515443] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3794.076135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.251752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3794.271480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3794.346298] Lustre: lustre-MDT0000: new disk, initializing [ 3794.413535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.430423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3798.213798] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3808.628644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.742669] Lustre: 112829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3808.796065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3808.801413] Lustre: Skipped 1 previous similar message [ 3808.874357] Lustre: lustre-MDT0001: new disk, initializing [ 3808.973871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3809.013686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3809.020865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3813.134797] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3822.931710] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3828.524806] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3835.807783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3836.115626] Lustre: lustre-OST0000: new disk, initializing [ 3836.118874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3836.124850] Lustre: 114800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3836.205291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.733258] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3843.666052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3843.677548] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3843.802210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3850.703833] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3855.346789] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3855.570298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3860.195757] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3860.389714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3864.546041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3864.556563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.568050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.574416] Lustre: Skipped 1 previous similar message [ 3864.585868] Lustre: Skipped 1 previous similar message [ 3868.639581] Lustre: server umount lustre-OST0000 complete [ 3874.793554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.801495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3878.300520] Lustre: server umount lustre-MDT0000 complete [ 3881.830458] LustreError: 112823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998368 with bad export cookie 7958276313880459531 [ 3881.834600] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3881.854113] LustreError: 112823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3882.231505] Lustre: server umount lustre-MDT0001 complete [ 3895.016905] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:12:59 (1787998379) [ 3898.872579] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3917.010670] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 3929.803925] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3964.913899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.214818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3965.259867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3965.358518] Lustre: lustre-MDT0000: new disk, initializing [ 3965.447955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.467672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3971.119575] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3982.866371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.051570] Lustre: 119702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3983.137759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3983.141545] Lustre: Skipped 1 previous similar message [ 3983.247828] Lustre: lustre-MDT0001: new disk, initializing [ 3983.352536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3983.400794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3983.417835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3988.629085] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3998.880769] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4006.084928] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.689765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.154292] Lustre: lustre-OST0000: new disk, initializing [ 4014.163714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4014.178117] Lustre: 121676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4014.303825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.796619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4019.809222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4019.897213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4021.208952] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4031.941195] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4037.198307] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4037.433379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4042.284991] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4042.582468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4045.793869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4045.798587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.812274] LustreError: Skipped 1 previous similar message [ 4045.824316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.834422] Lustre: Skipped 2 previous similar messages [ 4045.865831] Lustre: Skipped 2 previous similar messages [ 4050.760846] Lustre: server umount lustre-OST0000 complete [ 4056.035083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4056.040732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4056.046904] Lustre: Skipped 1 previous similar message [ 4061.153930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.166938] Lustre: Skipped 2 previous similar messages [ 4061.450067] Lustre: server umount lustre-MDT0000 complete [ 4065.034760] LustreError: 119693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998551 with bad export cookie 7958276313880460413 [ 4065.037248] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.043418] LustreError: 119693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4065.334739] Lustre: server umount lustre-MDT0001 complete [ 4074.885957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.308328] LustreError: 123892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.326826] LustreError: 123892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4075.391277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.100976] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4080.613673] LustreError: 123893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.707258] LustreError: 123892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.510595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.482376] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4101.552229] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4107.827218] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4115.419077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4121.923033] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4129.526677] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4140.080152] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4144.618068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.619585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4144.621066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4144.636279] Lustre: Skipped 1 previous similar message [ 4144.672764] Lustre: Skipped 1 previous similar message [ 4150.024788] Lustre: server umount lustre-OST0000 complete [ 4154.850143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.864818] Lustre: Skipped 1 previous similar message [ 4165.089483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4165.099947] Lustre: Skipped 7 previous similar messages [ 4168.159628] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4168.349187] Lustre: server umount lustre-MDT0000 complete [ 4170.219445] LustreError: 123888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.236654] LustreError: 123888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4172.502985] LustreError: 124629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998659 with bad export cookie 7958276313880461295 [ 4172.506045] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.519581] LustreError: 124629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4172.816710] Lustre: server umount lustre-MDT0001 complete [ 4183.827688] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 4186.382671] Key type lgssc unregistered [ 4186.633390] LNet: 127529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4186.652329] LNetError: 127529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4186.709348] LNet: Removed LNI 192.168.204.154@tcp [ 4187.589989] Key type .llcrypt unregistered [ 4187.592562] Key type ._llcrypt unregistered [ 4205.423800] Key type ._llcrypt registered [ 4205.426556] Key type .llcrypt registered [ 4205.520641] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 4221.783795] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 4223.050746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4223.077976] alg: No test for adler32 (adler32-zlib) [ 4224.306873] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 4224.849484] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 4226.657044] Key type lgssc registered [ 4228.058948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4268.327136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4268.346174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.574552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4269.598298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4269.665906] Lustre: lustre-MDT0000: new disk, initializing [ 4269.726314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.737876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4273.179181] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4283.302910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.412734] Lustre: 130683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4283.484674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4283.489308] Lustre: Skipped 1 previous similar message [ 4283.646943] Lustre: lustre-MDT0001: new disk, initializing [ 4283.796654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4283.839502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4283.856068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4288.105662] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4295.798582] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4300.812973] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4306.565080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.849317] Lustre: lustre-OST0000: new disk, initializing [ 4306.852828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4306.858322] Lustre: 132652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4306.948764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.107519] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4314.661168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4314.678445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4314.804572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4322.201838] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4326.859391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4327.025898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4331.555532] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4331.776691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4335.072636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4335.080499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.106604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.584652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4335.594626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.614817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.728398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.741412] Lustre: Skipped 1 previous similar message [ 4345.825633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4345.833147] Lustre: Skipped 1 previous similar message [ 4347.871990] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4348.023459] Lustre: server umount lustre-OST0000 complete [ 4355.552498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4355.564517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.600196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.402349] Lustre: server umount lustre-MDT0000 complete [ 4362.221907] LustreError: 130676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998848 with bad export cookie 16645464693659685892 [ 4362.231675] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.237561] LustreError: 130676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4362.699702] Lustre: server umount lustre-MDT0001 complete [ 4371.480319] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:20:56 (1787998856) [ 4377.645830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4378.160903] LustreError: 135063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.183135] LustreError: 135063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4378.227167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.958615] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4383.718929] LustreError: 135064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.812254] LustreError: 135063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.111701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.398542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4396.461726] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4403.934758] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4409.732447] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.982050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.997282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.999891] Lustre: Skipped 2 previous similar messages [ 4412.023957] Lustre: Skipped 3 previous similar messages [ 4418.367075] Lustre: server umount lustre-MDT0000 complete [ 4422.624285] LustreError: 135064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.628991] LustreError: 135064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4422.730278] LustreError: 135043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998909 with bad export cookie 16645464693659686774 [ 4422.732971] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.733894] LustreError: 135043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4422.907127] Lustre: server umount lustre-MDT0001 complete [ 4432.301217] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:21:57 (1787998917) [ 4438.674769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.303992] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4459.423531] LustreError: 137479:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 4485.151522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.976049] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4499.781844] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4503.784787] Lustre: server umount lustre-OST0000 complete [ 4512.173842] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:23:17 (1787998997) [ 4520.870745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4521.287200] LustreError: 138925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.370877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.063030] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4530.664862] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.683154] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4537.199133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.623815] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4551.484612] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4557.278621] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4564.190270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.723290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.728486] Lustre: Skipped 1 previous similar message [ 4571.306788] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4579.150958] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4593.641162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.653952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.657508] Lustre: Skipped 2 previous similar messages [ 4593.670974] Lustre: Skipped 4 previous similar messages [ 4598.754576] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.777645] LustreError: 138926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4598.852661] Lustre: server umount lustre-MDT0000 complete [ 4602.800095] LustreError: 138907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999089 with bad export cookie 16645464693659687439 [ 4602.803164] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4602.812598] LustreError: 138907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4603.140421] Lustre: server umount lustre-MDT0001 complete [ 4617.008576] Lustre: server umount lustre-OST0000 complete [ 4625.482096] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 06:25:10 (1787999110) [ 4632.581082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.977387] LustreError: 142600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.045079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.749521] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4644.123045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.056691] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4654.521874] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4659.078167] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4664.410778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.382772] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4678.153226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4679.806110] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4684.486660] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4684.678574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4689.604673] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4689.820317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4693.477633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4693.489934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.500516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4706.271185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4706.490712] Lustre: server umount lustre-OST0000 complete [ 4710.881628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4710.893212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.914468] Lustre: Skipped 1 previous similar message [ 4716.645559] Lustre: server umount lustre-MDT0000 complete [ 4719.073301] LustreError: 144918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.088229] LustreError: 144918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4720.599780] LustreError: 142582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999207 with bad export cookie 16645464693659688510 [ 4720.603132] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.608413] LustreError: 142582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4720.991308] Lustre: server umount lustre-MDT0001 complete [ 4730.233620] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:26:55 (1787999215) [ 4737.509641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4753.119618] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4758.239955] LustreError: 146759:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 4783.904792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.928602] Lustre: Skipped 2 previous similar messages [ 4790.539680] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4797.586268] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4804.347401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.474522] LustreError: 128905:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ffe989a1 x1874852536513408/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4804.858429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4808.687666] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4816.553410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.814465] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4827.356718] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4832.225648] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4836.238690] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4836.415668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4840.251158] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4840.401795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4842.464642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4842.478472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.494152] Lustre: Skipped 1 previous similar message [ 4842.499153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4842.501930] Lustre: Skipped 8 previous similar messages [ 4856.287329] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4856.459037] Lustre: server umount lustre-MDT0000 complete [ 4859.676762] LustreError: 146766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999346 with bad export cookie 16645464693659689532 [ 4859.691385] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4859.876478] LustreError: 148646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.890345] LustreError: 148646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4859.967936] Lustre: server umount lustre-MDT0001 complete [ 4865.665494] Lustre: server umount lustre-OST0000 complete [ 4873.202171] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 06:29:18 (1787999358) [ 4879.969400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4895.583767] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4900.705533] LustreError: 150954:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 4932.123857] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4939.098132] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4945.557510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4945.954843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4950.259463] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4959.428679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.345702] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4971.245730] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4977.254341] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4984.078331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4984.298569] Lustre: lustre-OST0001: new disk, initializing [ 4984.301692] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4984.307931] Lustre: 153825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4989.942905] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 4990.563835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4990.568458] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4990.657283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4998.046825] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5001.853507] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5002.029382] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5006.244441] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5006.472579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5010.912378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5010.921512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.938360] Lustre: Skipped 3 previous similar messages [ 5010.946530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5010.952901] Lustre: Skipped 9 previous similar messages [ 5014.491153] Lustre: server umount lustre-OST0000 complete [ 5023.527733] Lustre: server umount lustre-OST0001 complete [ 5033.484847] Lustre: server umount lustre-MDT0000 complete [ 5036.307975] LustreError: 150962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999522 with bad export cookie 16645464693659690484 [ 5036.321657] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.322248] LustreError: 150962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5036.577625] Lustre: server umount lustre-MDT0001 complete [ 5055.299564] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 06:32:20 (1787999540) [ 5056.545574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5058.011836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5059.426662] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 06:32:24 (1787999544) [ 5064.976914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5065.175515] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5065.193949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5065.312787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.321815] Lustre: Skipped 6 previous similar messages [ 5068.572631] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5074.860043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.925897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5074.960207] Lustre: 157382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5078.570876] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5083.794788] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5087.481558] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5092.354029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5092.500233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5097.875559] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5099.089808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5104.774687] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5108.752342] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5108.928866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5112.874762] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5113.049790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5119.462638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5119.471288] LustreError: Skipped 4 previous similar messages [ 5129.183416] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5129.318485] Lustre: server umount lustre-OST0000 complete [ 5149.546527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5150.891943] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.911542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.911853] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5154.834994] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5161.144575] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5165.212905] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5165.417019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5168.917940] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5169.101672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5176.513993] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96f8c4d9f000.ost_server_uuid 50 [ 5180.020808] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96f8c4d9f000.ost_server_uuid in FULL state after 0 sec [ 5180.187595] LustreError: 159598:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.54@tcp) returned error from blocking AST (req@00000000bfa0b0de x1874852536644864 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000909106bf/0xe70091f28ae46f4e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.54@tcp remote: 0xaff111a7fc12f12c expref: 5 pid: 157390 timeout: 5280 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5180.213484] LustreError: lustre-MDT0000: A client on nid 192.168.204.54@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5180.220810] LustreError: 157378:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.54@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000909106bf/0xe70091f28ae46f4e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.54@tcp remote: 0xaff111a7fc12f12c expref: 6 pid: 157390 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5186.027411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5186.036512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.054505] Lustre: Skipped 7 previous similar messages [ 5189.261183] Lustre: server umount lustre-OST0000 complete [ 5205.983405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5206.157568] Lustre: server umount lustre-MDT0000 complete [ 5206.498935] LustreError: 157394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5206.513033] LustreError: 157394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5208.958318] LustreError: 157374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999695 with bad export cookie 16645464693659691667 [ 5208.959104] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5208.964905] LustreError: 157374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5223.391674] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5223.647797] Lustre: server umount lustre-MDT0001 complete [ 5231.553871] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5233.711774] Key type lgssc unregistered [ 5234.005778] LNet: 163501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5234.019122] LNetError: 163501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5234.040699] LNet: Removed LNI 192.168.204.154@tcp [ 5234.824425] Key type .llcrypt unregistered [ 5234.828865] Key type ._llcrypt unregistered [ 5255.718793] Key type ._llcrypt registered [ 5255.720404] Key type .llcrypt registered [ 5255.792135] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5256.587391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5256.605021] alg: No test for adler32 (adler32-zlib) [ 5257.566340] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5257.764488] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5259.441512] Key type lgssc registered [ 5260.392799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5266.630652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5266.643167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.977590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.986598] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5278.027783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.178294] LustreError: 164893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.201037] LustreError: 164893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5278.367328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5282.492765] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5289.585776] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5294.310640] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5299.723445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5300.034658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.025162] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5311.014621] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5313.330133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5319.818772] Lustre: Failing over lustre-MDT0000 [ 5320.013184] Lustre: server umount lustre-MDT0000 complete [ 5323.746094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.746924] LustreError: 166731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.751561] Lustre: Skipped 2 previous similar messages [ 5323.761857] LustreError: 166731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5328.869706] LustreError: 164887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.887483] LustreError: 164887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5331.850308] LustreError: 164873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.54@tcp arrived at 1787999818 with bad export cookie 11445687834896200049 [ 5331.876322] LustreError: 164873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5333.988579] LustreError: 166731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.007665] LustreError: 166731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5336.951766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.042849] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5337.234786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.282991] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5340.648546] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5342.646136] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5342.653220] Lustre: lustre-MDT0000: Denying connection for new client 75e9b8fb-5464-4baa-b719-8ef19668d0b0 (at 192.168.204.54@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5342.712613] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5352.936571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.939420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5352.953540] Lustre: Skipped 1 previous similar message [ 5352.961023] Lustre: Skipped 1 previous similar message [ 5356.604714] Lustre: server umount lustre-OST0000 complete [ 5359.857209] LustreError: 168622:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5359.870717] Lustre: 167971:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5359.883671] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5359.898909] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5359.912412] LustreError: 167971:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bcb6ea05 x1874853619606528/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5359.932567] LustreError: 167971:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5359.942803] LustreError: 167971:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5359.949594] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 5360.161222] Lustre: server umount lustre-MDT0000 complete [ 5363.168769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.177744] LustreError: 164887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5363.408065] LustreError: 164874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999850 with bad export cookie 11445687834896200378 [ 5363.410742] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5363.425585] LustreError: 164874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5369.791221] Lustre: server umount lustre-MDT0001 complete [ 5379.265388] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5382.002427] Key type lgssc unregistered [ 5382.263532] LNet: 169378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5382.269228] LNetError: 169378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5382.279846] LNet: Removed LNI 192.168.204.154@tcp [ 5382.999440] Key type .llcrypt unregistered [ 5383.004394] Key type ._llcrypt unregistered [ 5405.289803] Key type ._llcrypt registered [ 5405.293662] Key type .llcrypt registered [ 5405.425310] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5406.772381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5406.806092] alg: No test for adler32 (adler32-zlib) [ 5407.900852] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5408.120104] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5409.791250] Key type lgssc registered [ 5410.740488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5417.779201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5417.808384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.156192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.275131] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5428.713506] LustreError: 170788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.736581] LustreError: 170788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5431.389323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5431.760640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5435.647111] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5441.628111] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5445.270274] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5450.363766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.664604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.347707] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5461.049915] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5463.992507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5472.229718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.237480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.253290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.280393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5474.283918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5474.285381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.309426] Lustre: Skipped 1 previous similar message [ 5477.443604] Lustre: server umount lustre-OST0000 complete [ 5482.978790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5482.985174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.992558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5486.574143] Lustre: server umount lustre-MDT0000 complete [ 5489.632516] LustreError: 171516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.655305] LustreError: 171516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5489.768776] LustreError: 170770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999976 with bad export cookie 12293812798780645046 [ 5489.770299] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.776058] LustreError: 170770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5490.069755] Lustre: server umount lustre-MDT0001 complete [ 5499.076994] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5501.194103] Key type lgssc unregistered [ 5501.435850] LNet: 174327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5501.441923] LNetError: 174327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5502.502890] LNet: Removed LNI 192.168.204.154@tcp [ 5503.172044] Key type .llcrypt unregistered [ 5503.176873] Key type ._llcrypt unregistered [ 5530.635174] Key type ._llcrypt registered [ 5530.640460] Key type .llcrypt registered [ 5530.754438] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5531.413757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5531.541219] alg: No test for adler32 (adler32-zlib) [ 5532.592146] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5532.782220] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5534.431187] Key type lgssc registered [ 5535.294445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5541.032408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5541.049073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.341146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.949521] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5551.592851] LustreError: 176009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.620702] LustreError: 176009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5553.714993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.952347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.193066] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5564.756914] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5570.097825] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5576.086074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5576.321673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.654031] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5586.610821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5588.880766] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5595.109992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.120828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.131074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5597.162939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5597.163164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5597.179068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.715839] Lustre: server umount lustre-OST0000 complete [ 5605.346407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5605.352152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.359880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.363247] Lustre: Skipped 1 previous similar message [ 5607.392259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5607.394310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.513133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.518481] Lustre: Skipped 1 previous similar message [ 5618.143505] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5618.291405] Lustre: server umount lustre-MDT0000 complete [ 5620.964806] LustreError: 175988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000107 with bad export cookie 4459617244657953302 [ 5620.974654] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5621.280522] Lustre: server umount lustre-MDT0001 complete [ 5630.444199] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5632.739545] Key type lgssc unregistered [ 5633.014781] LNet: 179544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5633.029330] LNetError: 179544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5633.050161] LNet: Removed LNI 192.168.204.154@tcp [ 5633.765388] Key type .llcrypt unregistered [ 5633.768944] Key type ._llcrypt unregistered [ 5652.477473] Key type ._llcrypt registered [ 5652.480758] Key type .llcrypt registered [ 5652.591444] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5653.405831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5653.422085] alg: No test for adler32 (adler32-zlib) [ 5654.439058] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5654.657192] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5656.351237] Key type lgssc registered [ 5657.320890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5665.645545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5665.670438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.927989] Lustre: *** cfs_fail_loc=135, val=0*** [ 5666.990804] LustreError: 181047:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5666.995630] LustreError: 181047:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -2 [ 5667.001874] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5667.001874] [ 5667.006715] LustreError: MGC192.168.204.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5667.016975] LustreError: 181009:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5667.023151] LustreError: 181009:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5667.030150] LustreError: 181009:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5667.034327] LustreError: 181009:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5667.097295] Lustre: server umount lustre-MDT0000 complete [ 5667.100489] LustreError: 181009:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5672.452095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.887245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.625986] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5683.177943] LustreError: 181418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.188034] LustreError: 181418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5683.276184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.578889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.046892] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5692.636214] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5696.994506] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5699.041924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5699.046847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.053867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.139224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.141172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.157807] Lustre: Skipped 1 previous similar message [ 5704.843185] Lustre: server umount lustre-MDT0000 complete [ 5707.869217] LustreError: 181399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000194 with bad export cookie 14334443033035441000 [ 5707.874895] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5707.880194] LustreError: 181399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5708.107918] Lustre: server umount lustre-MDT0001 complete [ 5715.719491] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5717.729532] Key type lgssc unregistered [ 5717.953894] LNet: 183739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5717.962091] LNetError: 183739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5717.977546] LNet: Removed LNI 192.168.204.154@tcp [ 5718.629711] Key type .llcrypt unregistered [ 5718.634151] Key type ._llcrypt unregistered [ 5736.153923] Key type ._llcrypt registered [ 5736.159808] Key type .llcrypt registered [ 5736.251193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5736.456289] Key type .llcrypt unregistered [ 5736.460090] Key type ._llcrypt unregistered [ 5749.563980] Key type ._llcrypt registered [ 5749.568553] Key type .llcrypt registered [ 5749.644489] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5750.522411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5750.556107] alg: No test for adler32 (adler32-zlib) [ 5751.549617] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5751.755845] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5753.415985] Key type lgssc registered [ 5754.235781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5759.790868] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5759.807369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5775.327627] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5780.448356] LustreError: 185852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 5807.071489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.174853] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5817.744907] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5822.099811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.389495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.423515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5824.978657] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5828.577262] LustreError: 186890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.587308] LustreError: 186890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5830.844676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.113054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5833.940324] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5839.932596] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5844.447948] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5846.393693] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5856.742860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.744622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5856.744851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5856.755857] Lustre: Skipped 1 previous similar message [ 5861.909066] Lustre: server umount lustre-OST0000 complete [ 5866.977357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.985305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5866.985909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5866.985914] Lustre: Skipped 1 previous similar message [ 5866.985956] Lustre: Skipped 1 previous similar message [ 5871.445860] Lustre: server umount lustre-MDT0000 complete [ 5872.103001] LustreError: 186895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.129589] LustreError: 186895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5874.454425] LustreError: 188512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000361 with bad export cookie 421614329421834866 [ 5874.456608] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.465556] LustreError: 188512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5874.724428] Lustre: server umount lustre-MDT0001 complete [ 5883.134252] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 5885.473130] Key type lgssc unregistered [ 5885.752880] LNet: 189816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.771849] LNetError: 189816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.794187] LNet: Removed LNI 192.168.204.154@tcp [ 5886.540813] Key type .llcrypt unregistered [ 5886.543234] Key type ._llcrypt unregistered [ 5907.194906] Key type ._llcrypt registered [ 5907.196896] Key type .llcrypt registered [ 5907.269703] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 5908.058452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5908.082923] alg: No test for adler32 (adler32-zlib) [ 5909.086728] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5909.291547] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 5910.999173] Key type lgssc registered [ 5911.869537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5917.679069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5917.699556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.051325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.401711] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5928.420216] LustreError: 191229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.438933] LustreError: 191229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5929.111647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.414192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5932.624874] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5938.350646] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5942.975205] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5948.315727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.588071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.199958] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5958.713491] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5963.948632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5966.836833] Lustre: Failing over lustre-MDT0000 [ 5967.069423] Lustre: server umount lustre-MDT0000 complete [ 5969.379952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.390493] LustreError: 193729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.391764] Lustre: Skipped 1 previous similar message [ 5974.497801] LustreError: 191956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.519122] LustreError: 191956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5979.618060] LustreError: 191229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5979.635301] LustreError: 191229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5983.920904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.013270] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5984.271437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5984.322238] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5985.006378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5987.601369] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 5989.363205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5989.379533] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5989.417692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5990.932616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5997.203411] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6009.828784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.829177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6009.846400] Lustre: Skipped 1 previous similar message [ 6009.848069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.857764] Lustre: Skipped 1 previous similar message [ 6014.951992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6014.964183] Lustre: Skipped 1 previous similar message [ 6015.706772] Lustre: server umount lustre-OST0000 complete [ 6020.065587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6020.067570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.074804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.085803] Lustre: Skipped 2 previous similar messages [ 6020.094707] Lustre: Skipped 1 previous similar message [ 6025.184872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6025.197264] Lustre: Skipped 1 previous similar message [ 6030.304941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6030.310503] Lustre: Skipped 1 previous similar message [ 6033.375443] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.481448] Lustre: server umount lustre-MDT0000 complete [ 6035.426454] LustreError: 191228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.436843] LustreError: 191228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6035.880912] LustreError: 191963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000522 with bad export cookie 11621196034440297859 [ 6035.882389] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.895358] LustreError: 191963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6036.165850] Lustre: server umount lustre-MDT0001 complete [ 6043.892614] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6045.970207] Key type lgssc unregistered [ 6046.176718] LNet: 196147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6046.182298] LNetError: 196147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6046.203499] LNet: Removed LNI 192.168.204.154@tcp [ 6046.834074] Key type .llcrypt unregistered [ 6046.838828] Key type ._llcrypt unregistered [ 6063.448766] Key type ._llcrypt registered [ 6063.450917] Key type .llcrypt registered [ 6063.526211] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6064.186407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6064.224031] alg: No test for adler32 (adler32-zlib) [ 6065.169509] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6065.336625] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6066.960650] Key type lgssc registered [ 6067.614187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6072.100526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6072.113969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.408634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.743624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6082.531734] LustreError: 197539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6082.544581] LustreError: 197539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6082.662402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6082.877076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6085.806192] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6090.967541] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6095.358789] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6100.444189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.706592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.480822] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6112.092325] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.130892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6119.667908] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6121.754565] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6136.116644] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6149.096416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.098090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6149.102580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6149.112314] Lustre: Skipped 1 previous similar message [ 6149.132880] LustreError: Skipped 1 previous similar message [ 6150.423530] Lustre: server umount lustre-OST0000 complete [ 6154.211473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.218350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6154.220958] Lustre: Skipped 1 previous similar message [ 6154.224247] Lustre: Skipped 1 previous similar message [ 6159.180278] Lustre: server umount lustre-MDT0000 complete [ 6159.328487] LustreError: 200029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.346601] LustreError: 200029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6161.979389] LustreError: 197517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000648 with bad export cookie 6404437461809653290 [ 6161.984950] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6161.990965] LustreError: 197517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6162.240358] Lustre: server umount lustre-MDT0001 complete [ 6170.073248] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6171.784699] Key type lgssc unregistered [ 6171.970567] LNet: 201400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6171.974209] LNetError: 201400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6171.983468] LNet: Removed LNI 192.168.204.154@tcp [ 6172.434610] Key type .llcrypt unregistered [ 6172.436085] Key type ._llcrypt unregistered [ 6183.640759] Key type ._llcrypt registered [ 6183.643477] Key type .llcrypt registered [ 6183.706362] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6184.157643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6184.223587] alg: No test for adler32 (adler32-zlib) [ 6185.156113] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6185.302945] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6186.927258] Key type lgssc registered [ 6187.568978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6192.283456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6192.300894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.579240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.109796] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6201.345079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.575512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6203.845117] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6208.411476] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.234091] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.321757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6217.600638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.628088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6221.872387] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6227.579384] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6232.947166] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6237.069735] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6246.579898] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6250.682120] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6262.754695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.761725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6262.762590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6262.770766] Lustre: Skipped 1 previous similar message [ 6267.803622] Lustre: server umount lustre-OST0000 complete [ 6272.997361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.001537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6273.006311] Lustre: Skipped 2 previous similar messages [ 6273.013829] Lustre: Skipped 1 previous similar message [ 6278.117647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6278.122687] Lustre: Skipped 1 previous similar message [ 6283.240906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6283.249401] Lustre: Skipped 1 previous similar message [ 6285.279212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6285.402635] Lustre: server umount lustre-MDT0000 complete [ 6288.153529] LustreError: 205573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000774 with bad export cookie 13639333528157274437 [ 6288.154115] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.160954] LustreError: 205573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6288.400503] Lustre: server umount lustre-MDT0001 complete [ 6297.131915] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6299.551518] Key type lgssc unregistered [ 6299.835764] LNet: 207570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6299.849677] LNetError: 207570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6299.864148] LNet: Removed LNI 192.168.204.154@tcp [ 6300.550377] Key type .llcrypt unregistered [ 6300.553213] Key type ._llcrypt unregistered [ 6317.947833] Key type ._llcrypt registered [ 6317.950972] Key type .llcrypt registered [ 6318.023425] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6318.708409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6318.738631] alg: No test for adler32 (adler32-zlib) [ 6319.755461] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6319.975916] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6321.655363] Key type lgssc registered [ 6322.650655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6328.316272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6328.347993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.726992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.912424] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6339.161841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.460306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6342.284155] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6346.872740] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6350.401175] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6354.892880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6355.133306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.145722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6359.633076] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6364.788949] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6374.881690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.886080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6374.896949] Lustre: Skipped 1 previous similar message [ 6374.910771] Lustre: Skipped 1 previous similar message [ 6380.013841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6380.019566] Lustre: Skipped 1 previous similar message [ 6380.938872] Lustre: server umount lustre-OST0000 complete [ 6385.636663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6385.646133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.655721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.192954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.194026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6389.848124] Lustre: server umount lustre-MDT0000 complete [ 6392.468755] LustreError: 208942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000879 with bad export cookie 6834565111628753871 [ 6392.475883] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.478976] LustreError: 208942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6392.684605] Lustre: server umount lustre-MDT0001 complete [ 6399.163317] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6400.832755] Key type lgssc unregistered [ 6401.050191] LNet: 212642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6401.054331] LNetError: 212642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6401.063770] LNet: Removed LNI 192.168.204.154@tcp [ 6401.589603] Key type .llcrypt unregistered [ 6401.591340] Key type ._llcrypt unregistered [ 6412.741927] Key type ._llcrypt registered [ 6412.743566] Key type .llcrypt registered [ 6412.818349] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6413.232020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6413.275037] alg: No test for adler32 (adler32-zlib) [ 6414.189034] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6414.318486] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6415.919166] Key type lgssc registered [ 6416.642463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6421.609206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6421.622361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.912036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6425.693174] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6431.451954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.701522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.270757] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6438.792256] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6442.648618] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6446.971242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6447.231354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.233063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6450.732878] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6455.488392] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6463.968769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6463.972328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.979967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.041205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.042539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.045523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.104756] Lustre: server umount lustre-OST0000 complete [ 6472.674294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6472.678475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.692149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6472.695339] Lustre: Skipped 1 previous similar message [ 6477.281208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.288759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.957475] Lustre: server umount lustre-MDT0000 complete [ 6481.314585] LustreError: 215731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000967 with bad export cookie 17852403581413377305 [ 6481.315630] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.321276] LustreError: 215731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6481.528125] Lustre: server umount lustre-MDT0001 complete [ 6488.388598] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6490.139685] Key type lgssc unregistered [ 6490.361337] LNet: 217569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6490.368523] LNetError: 217569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6490.384380] LNet: Removed LNI 192.168.204.154@tcp [ 6490.889236] Key type .llcrypt unregistered [ 6490.891376] Key type ._llcrypt unregistered [ 6502.349935] Key type ._llcrypt registered [ 6502.351938] Key type .llcrypt registered [ 6502.453207] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6503.109917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6503.129791] alg: No test for adler32 (adler32-zlib) [ 6504.075635] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6504.237666] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6505.848537] Key type lgssc registered [ 6506.483117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6510.904940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6510.919094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.183785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.483505] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6519.287905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.483373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6521.495153] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6525.740924] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6529.004964] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6533.298532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.539253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.616078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6537.149122] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6541.847316] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6547.769045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.818534] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6547.974432] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6551.878552] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6556.575299] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6557.492480] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6568.577820] Lustre: Permanently deactivating lustre-OST0001 [ 6568.589314] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6578.202356] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6585.217403] Lustre: Permanently reactivating lustre-OST0001 [ 6585.237514] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6585.242141] Lustre: Skipped 2 previous similar messages [ 6585.629925] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.641846] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6585.653827] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6585.660774] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6585.697151] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6596.068353] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6596.069426] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.074510] LustreError: Skipped 1 previous similar message [ 6596.084100] Lustre: Skipped 1 previous similar message [ 6596.085046] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6596.097061] Lustre: Skipped 1 previous similar message [ 6600.276697] Lustre: server umount lustre-OST0001 complete [ 6601.697657] LustreError: 220630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.714779] LustreError: 220630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6603.232569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6603.239328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.247363] Lustre: Skipped 1 previous similar message [ 6603.251813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6606.304294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6606.309407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.317588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6606.821764] LustreError: 220630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.836048] LustreError: 220630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6609.205256] Lustre: server umount lustre-OST0000 complete [ 6611.936556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6611.939695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.947378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.956143] Lustre: Skipped 1 previous similar message [ 6611.961309] Lustre: Skipped 1 previous similar message [ 6617.059052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6617.063245] Lustre: Skipped 2 previous similar messages [ 6617.834127] Lustre: server umount lustre-MDT0000 complete [ 6620.175588] LustreError: 218794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788001106 with bad export cookie 3875191516992182360 [ 6620.183292] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.187652] LustreError: 218794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6620.420542] Lustre: server umount lustre-MDT0001 complete [ 6627.158624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6628.927158] Key type lgssc unregistered [ 6629.118092] LNet: 223904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6629.124413] LNetError: 223904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6629.133367] LNet: Removed LNI 192.168.204.154@tcp [ 6629.681763] Key type .llcrypt unregistered [ 6629.683530] Key type ._llcrypt unregistered [ 6653.657556] Key type ._llcrypt registered [ 6653.659294] Key type .llcrypt registered [ 6653.724528] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6654.294804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6654.316191] alg: No test for adler32 (adler32-zlib) [ 6655.216154] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6655.344803] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6656.951179] Key type lgssc registered [ 6657.573444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6661.767249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6661.778281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.942937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6662.962988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6663.077625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.551523] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6670.466658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.527466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6670.552484] Lustre: 226087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6670.729317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6673.055850] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6677.411390] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6680.585716] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6684.175308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6684.272715] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6684.383878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.155897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6687.411348] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6691.489683] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6694.537977] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6698.354709] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6708.476966] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6717.525844] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6725.548399] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6746.587739] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6746.593395] Lustre: Skipped 2 previous similar messages [ 6786.915368] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6786.922192] Lustre: Skipped 4 previous similar messages [ 6853.937067] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6853.941434] Lustre: Skipped 7 previous similar messages [ 6869.476254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6869.478266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6869.484420] Lustre: Skipped 1 previous similar message [ 6869.492473] Lustre: Skipped 1 previous similar message [ 6874.426511] Lustre: server umount lustre-OST0000 complete [ 6879.713661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.715634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.724406] Lustre: Skipped 1 previous similar message [ 6879.732350] Lustre: Skipped 1 previous similar message [ 6882.670360] Lustre: server umount lustre-MDT0000 complete [ 6884.642591] LustreError: 226079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788001371 with bad export cookie 2127264349218724044 [ 6884.647025] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6884.649871] LustreError: 226079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6884.829714] Lustre: server umount lustre-MDT0001 complete [ 6890.476877] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6891.942578] Key type lgssc unregistered [ 6892.121853] LNet: 230913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6892.125800] LNetError: 230913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6892.136798] LNet: Removed LNI 192.168.204.154@tcp [ 6892.634878] Key type .llcrypt unregistered [ 6892.636842] Key type ._llcrypt unregistered [ 6906.293552] Key type ._llcrypt registered [ 6906.294850] Key type .llcrypt registered [ 6906.361401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6906.926366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6906.938922] alg: No test for adler32 (adler32-zlib) [ 6907.835186] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6907.952886] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6909.551168] Key type lgssc registered [ 6910.243903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6914.380481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6914.390695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.608852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.527894] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6921.500844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.682591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6923.573067] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6927.519308] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6930.617534] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6934.580278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.806693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.271636] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6940.979188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6942.633208] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6961.633495] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6961.637566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.648124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6962.658977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6962.660734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6962.662809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.030512] Lustre: server umount lustre-OST0000 complete [ 6967.777565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6967.778720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.782904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6967.791657] Lustre: Skipped 1 previous similar message [ 6967.799682] Lustre: Skipped 2 previous similar messages [ 6972.298545] Lustre: server umount lustre-MDT0000 complete [ 6972.900705] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.911147] LustreError: 234889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6974.501828] LustreError: 232284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788001461 with bad export cookie 7125246229678163638 [ 6974.506085] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6974.509419] LustreError: 232284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6974.672644] Lustre: server umount lustre-MDT0001 complete [ 6980.581243] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6982.062860] Key type lgssc unregistered [ 6982.251692] LNet: 235989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6982.256589] LNetError: 235989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6982.267609] LNet: Removed LNI 192.168.204.154@tcp [ 6982.662714] Key type .llcrypt unregistered [ 6982.664707] Key type ._llcrypt unregistered [ 6994.509596] Key type ._llcrypt registered [ 6994.511079] Key type .llcrypt registered [ 6994.556099] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6994.978640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6995.022097] alg: No test for adler32 (adler32-zlib) [ 6995.952649] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6996.097329] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6997.711549] Key type lgssc registered [ 6998.401393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7264.661712] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7266.136936] Key type lgssc unregistered [ 7266.304959] LNet: 237995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7266.309478] LNetError: 237995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7266.321596] LNet: Removed LNI 192.168.204.154@tcp [ 7266.740526] Key type .llcrypt unregistered [ 7266.744221] Key type ._llcrypt unregistered [ 7280.568220] Key type ._llcrypt registered [ 7280.571355] Key type .llcrypt registered [ 7280.634894] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7289.680207] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7290.118499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7290.154206] alg: No test for adler32 (adler32-zlib) [ 7291.130174] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 7291.273123] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7292.887253] Key type lgssc registered [ 7293.477582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7307.275982] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7012 sec ========== 07:09:52 (1788001792) [ 7308.041258] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:09:53 (1788001793) === [ 7309.488978] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:09:55 (1788001795) === [ 7328.784334] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7330.034881] Key type lgssc unregistered [ 7330.190608] LNet: 243204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7330.195329] LNetError: 243204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7330.208345] LNet: Removed LNI 192.168.204.154@tcp [ 7330.597779] Key type .llcrypt unregistered [ 7330.600552] Key type ._llcrypt unregistered