[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 545467515 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003087] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008506] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.012082] Yama: becoming mindful. [ 0.013061] SELinux: Initializing. [ 0.014129] *** VALIDATE selinux *** [ 0.023353] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028697] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030074] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032140] *** VALIDATE tmpfs *** [ 0.034539] *** VALIDATE proc *** [ 0.036070] *** VALIDATE cgroup *** [ 0.037014] *** VALIDATE cgroup2 *** [ 0.039147] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040189] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042046] Spectre V2 : User space: Vulnerable [ 0.043017] Speculative Store Bypass: Vulnerable [ 0.046749] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.048960] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049781] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050025] ... version: 2 [ 0.051017] ... bit width: 48 [ 0.052016] ... generic registers: 4 [ 0.053015] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055016] ... fixed-purpose events: 3 [ 0.056014] ... event mask: 000000070000000f [ 0.057294] rcu: Hierarchical SRCU implementation. [ 0.059746] smp: Bringing up secondary CPUs ... [ 0.060568] x86: Booting SMP configuration: [ 0.061022] .... node #0, CPUs: #1 #2 #3 [ 0.075192] smp: Brought up 1 node, 4 CPUs [ 0.077014] smpboot: Max logical packages: 1 [ 0.078106] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.169257] node 0 deferred pages initialised in 89ms [ 0.172272] devtmpfs: initialized [ 0.173285] x86/mm: Memory block size: 128MB [ 0.176263] gcov: version magic: 0x41383552 [ 0.178280] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.185331] pinctrl core: initialized pinctrl subsystem [ 0.187246] [ 0.187811] ************************************************************* [ 0.190017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192020] ** ** [ 0.195017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.197017] ** ** [ 0.199020] ** This means that this kernel is built to expose internal ** [ 0.202014] ** IOMMU data structures, which may compromise security on ** [ 0.204014] ** your system. ** [ 0.206019] ** ** [ 0.208018] ** If you see this message and you are not debugging the ** [ 0.210018] ** kernel, report this immediately to your vendor! ** [ 0.213015] ** ** [ 0.215012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.217018] ************************************************************* [ 0.219723] NET: Registered protocol family 16 [ 0.221471] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.224060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.226064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230123] cpuidle: using governor menu [ 0.231739] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.232650] PCI: Using configuration type 1 for base access [ 0.234140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.245138] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.248034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.251186] cryptd: max_cpu_qlen set to 1000 [ 0.254183] ACPI: Added _OSI(Module Device) [ 0.256023] ACPI: Added _OSI(Processor Device) [ 0.257014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.259017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.263179] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.270544] ACPI: Interpreter enabled [ 0.272076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273021] ACPI: Using IOAPIC for interrupt routing [ 0.275124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.278415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.288791] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.291062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.294032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.297131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.302439] acpiphp: Slot [2] registered [ 0.304181] acpiphp: Slot [5] registered [ 0.306195] acpiphp: Slot [6] registered [ 0.307160] acpiphp: Slot [7] registered [ 0.308162] acpiphp: Slot [8] registered [ 0.310150] acpiphp: Slot [9] registered [ 0.311162] acpiphp: Slot [10] registered [ 0.313159] acpiphp: Slot [3] registered [ 0.314145] acpiphp: Slot [4] registered [ 0.316123] acpiphp: Slot [11] registered [ 0.317107] acpiphp: Slot [12] registered [ 0.319119] acpiphp: Slot [13] registered [ 0.320157] acpiphp: Slot [14] registered [ 0.323175] acpiphp: Slot [15] registered [ 0.324111] acpiphp: Slot [16] registered [ 0.326106] acpiphp: Slot [17] registered [ 0.327093] acpiphp: Slot [18] registered [ 0.329161] acpiphp: Slot [19] registered [ 0.330210] acpiphp: Slot [20] registered [ 0.332128] acpiphp: Slot [21] registered [ 0.333094] acpiphp: Slot [22] registered [ 0.334224] acpiphp: Slot [23] registered [ 0.336103] acpiphp: Slot [24] registered [ 0.338123] acpiphp: Slot [25] registered [ 0.340178] acpiphp: Slot [26] registered [ 0.341143] acpiphp: Slot [27] registered [ 0.343109] acpiphp: Slot [28] registered [ 0.344106] acpiphp: Slot [29] registered [ 0.345158] acpiphp: Slot [30] registered [ 0.347112] acpiphp: Slot [31] registered [ 0.348084] PCI host bridge to bus 0000:00 [ 0.350020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.352031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.355019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357051] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.360030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.362026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.364195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.368192] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371443] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.380015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.385515] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396670] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399803] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.402072] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.405874] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.422977] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.428023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.434051] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.440020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.445018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.468240] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.474030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.484019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.505031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517088] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.524018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.557000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.569018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.586022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.595795] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.607027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.613015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.650515] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.660028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.667021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.686042] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.698496] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.701462] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.704558] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706389] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.708292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.715144] iommu: Default domain type: Passthrough [ 0.717462] SCSI subsystem initialized [ 0.718148] ACPI: bus type USB registered [ 0.719071] usbcore: registered new interface driver usbfs [ 0.720097] usbcore: registered new interface driver hub [ 0.722113] usbcore: registered new device driver usb [ 0.723159] pps_core: LinuxPPS API ver. 1 registered [ 0.725010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727074] PTP clock support registered [ 0.728211] EDAC MC: Ver: 3.0.0 [ 0.730144] PCI: Using ACPI for IRQ routing [ 0.731841] NetLabel: Initializing [ 0.733020] NetLabel: domain hash size = 128 [ 0.733978] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.735074] NetLabel: unlabeled traffic allowed by default [ 0.737149] vgaarb: loaded [ 0.738217] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.739010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744018] clocksource: Switched to clocksource kvm-clock [ 0.862058] VFS: Disk quotas dquot_6.6.0 [ 0.863584] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.866168] *** VALIDATE ramfs *** [ 0.867445] *** VALIDATE hugetlbfs *** [ 0.869041] pnp: PnP ACPI init [ 0.871412] pnp: PnP ACPI: found 6 devices [ 0.890199] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.893273] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895314] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.897513] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.900037] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.902503] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.905658] NET: Registered protocol family 2 [ 0.909272] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.915536] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.919721] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.924975] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.928683] TCP: Hash tables configured (established 65536 bind 65536) [ 0.931577] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.936605] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.939720] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.943738] NET: Registered protocol family 1 [ 0.948312] RPC: Registered named UNIX socket transport module. [ 0.950814] RPC: Registered udp transport module. [ 0.952500] RPC: Registered tcp transport module. [ 0.954101] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.956566] NET: Registered protocol family 44 [ 0.957877] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.960082] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.962531] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.965677] PCI: CLS 0 bytes, default 64 [ 0.967533] Unpacking initramfs... [ 2.351114] debug: unmapping init [mem 0xffff9a467cc54000-0xffff9a467ffbffff] [ 2.353799] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.355430] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.357381] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.845164] Initialise system trusted keyrings [ 2.847183] Key type blacklist registered [ 2.849227] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.858373] zbud: loaded [ 2.861721] *** VALIDATE nfs *** [ 2.863212] *** VALIDATE nfs4 *** [ 2.865089] pstore: using deflate compression [ 2.871885] Platform Keyring initialized [ 2.986089] NET: Registered protocol family 38 [ 2.987933] Key type asymmetric registered [ 2.989546] Asymmetric key parser 'x509' registered [ 2.991689] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.995459] io scheduler mq-deadline registered [ 2.997463] io scheduler kyber registered [ 2.999476] io scheduler bfq registered [ 3.001632] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.005219] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.007864] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.010810] ACPI: Power Button [PWRF] [ 3.016621] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025896] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.040672] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.048836] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.068822] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.103124] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.143117] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.148277] Non-volatile memory driver v1.3 [ 3.150419] Linux agpgart interface v0.103 [ 3.184796] virtio_blk virtio1: [vda] 149376 512-byte logical blocks (76.5 MB/72.9 MiB) [ 3.188073] vda: detected capacity change from 0 to 76480512 [ 3.202908] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.205568] vdb: detected capacity change from 0 to 1073741824 [ 3.223956] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.227163] vdc: detected capacity change from 0 to 2621440000 [ 3.242398] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.245343] vdd: detected capacity change from 0 to 2621440000 [ 3.257991] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.260971] vde: detected capacity change from 0 to 4294967296 [ 3.275374] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.277612] vdf: detected capacity change from 0 to 4294967296 [ 3.283454] libphy: Fixed MDIO Bus: probed [ 3.290260] usbcore: registered new interface driver usbserial_generic [ 3.292088] usbserial: USB Serial support registered for generic [ 3.293769] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.296795] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.298594] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.300544] mousedev: PS/2 mouse device common for all mice [ 3.302605] rtc_cmos 00:05: RTC can wake from S4 [ 3.304950] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.305113] rtc_cmos 00:05: registered as rtc0 [ 3.308837] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.310904] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.311846] intel_pstate: CPU model not supported [ 3.316121] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.318594] hid: raw HID events driver (C) Jiri Kosina [ 3.321068] usbcore: registered new interface driver usbhid [ 3.322192] usbhid: USB HID core driver [ 3.323143] drop_monitor: Initializing network drop monitor service [ 3.325198] Initializing XFRM netlink socket [ 3.326989] NET: Registered protocol family 10 [ 3.329602] Segment Routing with IPv6 [ 3.330558] NET: Registered protocol family 17 [ 3.332124] mpls_gso: MPLS GSO support [ 3.337095] RAS: Correctable Errors collector initialized. [ 3.339401] AVX version of gcm_enc/dec engaged. [ 3.340648] AES CTR mode by8 optimization enabled [ 3.408049] sched_clock: Marking stable (3408013709, 0)->(4395629629, -987615920) [ 3.411771] registered taskstats version 1 [ 3.413632] Loading compiled-in X.509 certificates [ 3.415580] zswap: loaded using pool lzo/zbud [ 3.437553] Key type big_key registered [ 3.449570] Key type encrypted registered [ 3.451028] ima: No TPM chip found, activating TPM-bypass! [ 3.453214] ima: Allocated hash algorithm: sha1 [ 3.455073] ima: No architecture policies found [ 3.456834] evm: Initialising EVM extended attributes: [ 3.458749] evm: security.selinux [ 3.460030] evm: security.ima [ 3.461149] evm: security.capability [ 3.462514] evm: HMAC attrs: 0x1 [ 3.464857] rtc_cmos 00:05: setting system clock to 2026-08-22 10:08:21 UTC (1787393301) [ 3.471569] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.474717] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.485403] Write protecting the kernel read-only data: 28672k [ 3.488776] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.491811] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 3.527205] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.536533] systemd[1]: Detected virtualization kvm. [ 3.538057] systemd[1]: Detected architecture x86-64. [ 3.539996] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.571493] systemd[1]: No hostname configured. [ 3.572669] systemd[1]: Set hostname to . [ 3.574157] random: systemd: uninitialized urandom read (16 bytes read) [ 3.575659] systemd[1]: Initializing machine ID from random generator. [ 3.694736] random: systemd: uninitialized urandom read (16 bytes read) [ 3.696953] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.700626] random: systemd: uninitialized urandom read (16 bytes read) [ 3.703545] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.707493] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.358907] device-mapper: uevent: version 1.0.3 [ 4.361542] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK [ 5.155030] random: fast init done ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.222623] virtio_net virtio0 ens2: renamed from eth0 [ 5.392746] scsi host0: ata_piix [ 5.412735] scsi host1: ata_piix [ 5.414522] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.416784] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.135783] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.971393] random: crng init done [ 9.972449] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.554748] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.826588] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.064771] SELinux: Disabled at runtime. [ 12.130115] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.137772] systemd[1]: Detected virtualization kvm. [ 12.139351] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.613464] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.617029] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.626342] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.630273] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.632828] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.640502] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.667600] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ 12.734848] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.643228] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.131757] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.149124] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.555453] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.648826] EDAC sbridge: Ver: 1.1.2 [ 18.436421] Key type dns_resolver registered [ 18.989889] NFS: Registering the id_resolver key type [ 18.996967] Key type id_resolver registered [ 19.000344] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-server login: [ 73.225108] hrtimer: interrupt took 2866349 ns [ 76.518279] libcfs: loading out-of-tree module taints kernel. [ 76.605516] Key type ._llcrypt registered [ 76.611404] Key type .llcrypt registered [ 76.702206] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 94.571924] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 96.150663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.177391] alg: No test for adler32 (adler32-zlib) [ 97.654271] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 98.628923] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 100.432223] Key type lgssc registered [ 102.295051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.225657] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.494294] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 179.011563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.068222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.307658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.333690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.397633] Lustre: lustre-MDT0000: new disk, initializing [ 180.458555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.472513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.504926] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.621240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.774677] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 197.890963] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 197.908146] Lustre: Skipped 1 previous similar message [ 198.059450] Lustre: lustre-MDT0001: new disk, initializing [ 198.159606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.212441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 198.238806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 202.599607] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.983648] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 216.652827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.041901] Lustre: lustre-OST0000: new disk, initializing [ 217.048257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.065623] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.125191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.533542] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.903670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 226.926958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 227.037183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 239.469821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.659755] Lustre: lustre-OST0001: new disk, initializing [ 239.664058] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.671732] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.751670] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.254578] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.970362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.010419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.175927] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.757697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.347393] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.837118] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing check_logdir /tmp/testlogs/ [ 276.938323] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing yml_node [ 280.913598] Lustre: DEBUG MARKER: Client: 2.17.54.107 [ 283.083923] Lustre: DEBUG MARKER: MDS: 2.17.54.107 [ 285.487684] Lustre: DEBUG MARKER: OSS: 2.17.54.107 [ 286.918292] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 06:13:03 EDT 2026 [ 302.953259] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 304.216333] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.998810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 321.009620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.041542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.016967] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.017547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.031339] Lustre: Skipped 2 previous similar messages [ 322.041624] Lustre: Skipped 3 previous similar messages [ 324.930402] Lustre: server umount lustre-MDT0000 complete [ 328.090711] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393626 with bad export cookie 7220868803753308162 [ 328.095175] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.097286] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 332.257039] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.260625] LustreError: 9812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.279787] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.291319] LustreError: 9812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 332.300012] Lustre: Skipped 1 previous similar message [ 333.281596] LustreError: 9812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.308327] LustreError: 9812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 334.428872] Lustre: server umount lustre-MDT0001 complete [ 338.744543] Lustre: server umount lustre-OST0000 complete [ 343.409514] Lustre: server umount lustre-OST0001 complete [ 350.828520] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 361.708725] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 396.710508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.996630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 397.021199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 397.110439] Lustre: lustre-MDT0000: new disk, initializing [ 397.192219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 397.207057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 401.612119] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 412.426780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.554780] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.626183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.631510] Lustre: Skipped 1 previous similar message [ 412.737321] Lustre: lustre-MDT0001: new disk, initializing [ 412.840100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.899673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.909544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 416.085290] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 424.970690] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 430.783467] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 438.343291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.558299] Lustre: lustre-OST0000: new disk, initializing [ 438.561946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 438.567488] Lustre: 17041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 438.641091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.557360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 440.562510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 440.690846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.411865] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 453.065589] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 457.238713] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 457.457610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 462.631194] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 462.934527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 466.406528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.411848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.423414] LustreError: Skipped 1 previous similar message [ 466.424189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.424193] Lustre: Skipped 2 previous similar messages [ 466.481017] Lustre: Skipped 2 previous similar messages [ 470.940662] Lustre: server umount lustre-OST0000 complete [ 476.641615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.648284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.667413] Lustre: Skipped 1 previous similar message [ 476.683661] Lustre: Skipped 2 previous similar messages [ 481.757519] Lustre: server umount lustre-MDT0000 complete [ 481.767340] LustreError: 15079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.780539] LustreError: 15079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 485.405617] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393783 with bad export cookie 7220868803753309695 [ 485.422102] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.423933] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 485.824120] Lustre: server umount lustre-MDT0001 complete [ 489.033507] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:16:25 (1787393785) [ 493.927193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.299269] LustreError: 19306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.344523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.510943] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 499.689863] LustreError: 19307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.794399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.019233] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 517.937745] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 523.010876] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 529.509739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.257353] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 542.475167] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 553.442890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.455443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.466175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 555.817338] Lustre: server umount lustre-OST0000 complete [ 565.815832] Lustre: server umount lustre-MDT0000 complete [ 568.873423] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393866 with bad export cookie 7220868803753310577 [ 568.881364] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.885806] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 569.355363] Lustre: server umount lustre-MDT0001 complete [ 579.570685] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 582.346321] Key type lgssc unregistered [ 582.632642] LNet: 22838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.641874] LNetError: 22838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 583.655322] LNet: Removed LNI 192.168.201.113@tcp [ 584.740168] Key type .llcrypt unregistered [ 584.742084] Key type ._llcrypt unregistered [ 606.409239] Key type ._llcrypt registered [ 606.411473] Key type .llcrypt registered [ 606.513311] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 607.432291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.538798] alg: No test for adler32 (adler32-zlib) [ 608.593435] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 608.981774] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 610.656210] Key type lgssc registered [ 611.924525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 618.645551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 618.665775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.074893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.952902] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 630.247605] LustreError: 24249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.268778] LustreError: 24249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 631.314408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.618650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 635.908640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 643.698789] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 650.219472] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 657.497842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 657.790539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.583989] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 666.010628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 671.204440] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 686.563204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 686.567666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.568728] LustreError: Skipped 1 previous similar message [ 686.570091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 686.578819] Lustre: Skipped 1 previous similar message [ 690.383285] Lustre: server umount lustre-OST0000 complete [ 696.804167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.829459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.841128] Lustre: Skipped 2 previous similar messages [ 700.260771] Lustre: server umount lustre-MDT0000 complete [ 701.929068] LustreError: 24976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.945419] LustreError: 24976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 704.183604] LustreError: 24230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394002 with bad export cookie 9784488125764802349 [ 704.190954] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.193859] LustreError: 24230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 704.543502] Lustre: server umount lustre-MDT0001 complete [ 715.864396] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 718.151648] Key type lgssc unregistered [ 718.407627] LNet: 28229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.412258] LNetError: 28229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.431827] LNet: Removed LNI 192.168.201.113@tcp [ 719.264900] Key type .llcrypt unregistered [ 719.266625] Key type ._llcrypt unregistered [ 743.109926] Key type ._llcrypt registered [ 743.111536] Key type .llcrypt registered [ 743.274657] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 744.495845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 744.512958] alg: No test for adler32 (adler32-zlib) [ 745.488364] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 745.689667] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 747.354082] Key type lgssc registered [ 748.981724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.784043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.815415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.551393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.042666] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 769.000848] LustreError: 29642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.024988] LustreError: 29642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 772.070546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.307518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 775.962918] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 783.432232] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 789.419446] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 802.066969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.385437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.489984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 807.077955] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 814.690522] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 828.385778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.398141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.417763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 828.447172] Lustre: Skipped 1 previous similar message [ 828.478038] Lustre: Skipped 1 previous similar message [ 833.507677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 833.518189] Lustre: Skipped 1 previous similar message [ 834.198188] Lustre: server umount lustre-OST0000 complete [ 838.626645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.629306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.648469] Lustre: Skipped 1 previous similar message [ 843.755552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 843.772105] Lustre: Skipped 2 previous similar messages [ 844.334700] Lustre: server umount lustre-MDT0000 complete [ 847.862871] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394145 with bad export cookie 14856555929556549403 [ 847.864925] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 847.881808] LustreError: 29623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 848.285722] Lustre: server umount lustre-MDT0001 complete [ 857.383077] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 859.928693] Key type lgssc unregistered [ 860.157737] LNet: 33538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.165269] LNetError: 33538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.174854] LNet: Removed LNI 192.168.201.113@tcp [ 860.889884] Key type .llcrypt unregistered [ 860.891871] Key type ._llcrypt unregistered [ 883.149848] Key type ._llcrypt registered [ 883.154705] Key type .llcrypt registered [ 883.233816] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 884.581349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 884.794619] alg: No test for adler32 (adler32-zlib) [ 885.842425] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 886.161201] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 887.816294] Key type lgssc registered [ 888.680964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.489587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.511216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.907536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.807389] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 905.187418] LustreError: 34951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.210975] LustreError: 34951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 907.675023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.937080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 911.594040] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 918.875149] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 924.761138] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 930.688975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 930.947621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.931747] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 940.245887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 942.825645] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 952.804588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.821234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.849377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.838707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 953.853595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.869456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.880936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.898274] Lustre: Skipped 1 previous similar message [ 958.163085] Lustre: server umount lustre-OST0000 complete [ 964.068079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 964.074305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.088052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.116237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.681529] Lustre: server umount lustre-MDT0000 complete [ 971.247376] LustreError: 34951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.268370] LustreError: 34951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 971.500744] LustreError: 34931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394269 with bad export cookie 10115011447998918804 [ 971.515278] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.814903] Lustre: server umount lustre-MDT0001 complete [ 983.080558] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 985.470393] Key type lgssc unregistered [ 985.766656] LNet: 38485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.772361] LNetError: 38485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.788656] LNet: Removed LNI 192.168.201.113@tcp [ 986.472421] Key type .llcrypt unregistered [ 986.473970] Key type ._llcrypt unregistered [ 1010.786830] Key type ._llcrypt registered [ 1010.794425] Key type .llcrypt registered [ 1010.880858] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1011.943456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1011.987398] alg: No test for adler32 (adler32-zlib) [ 1013.075182] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1013.312154] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1014.968183] Key type lgssc registered [ 1015.938798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1024.604621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1024.628844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.147645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.969570] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1036.787131] LustreError: 39900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.816198] LustreError: 39900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1039.168961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.555260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.197534] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1052.450850] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1059.043777] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1066.212941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1066.533679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.674763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1073.680574] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1081.762969] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1092.069391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.078558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.092873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1094.028743] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1095.138590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1095.142288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.147366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.169013] Lustre: Skipped 1 previous similar message [ 1099.112103] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1104.228144] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1104.237519] Lustre: Skipped 2 previous similar messages [ 1105.376473] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1105.567781] Lustre: server umount lustre-OST0000 complete [ 1126.885568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.903686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.921401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.924592] Lustre: Skipped 2 previous similar messages [ 1130.981766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.618353] Lustre: server umount lustre-MDT0000 complete [ 1135.147123] LustreError: 39900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.161521] LustreError: 39900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1136.711908] LustreError: 39882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394434 with bad export cookie 4276084346959995077 [ 1136.717103] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.726549] LustreError: 39882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.185812] Lustre: server umount lustre-MDT0001 complete [ 1156.078425] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1158.790689] Key type lgssc unregistered [ 1159.118549] LNet: 43611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.127385] LNetError: 43611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.158399] LNet: Removed LNI 192.168.201.113@tcp [ 1159.885395] Key type .llcrypt unregistered [ 1159.891038] Key type ._llcrypt unregistered [ 1185.128658] Key type ._llcrypt registered [ 1185.131393] Key type .llcrypt registered [ 1185.249498] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1186.327976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1186.361273] alg: No test for adler32 (adler32-zlib) [ 1187.485342] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1187.718456] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1189.466042] Key type lgssc registered [ 1190.451616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1198.890403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1198.903940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.260913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.393880] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1210.850039] LustreError: 45024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.876671] LustreError: 45024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1212.713788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.043612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1217.512783] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1225.165959] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1231.969765] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1241.409399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.048210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.121778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1250.257465] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1258.684569] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1266.061828] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1268.711865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.732514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1268.738143] Lustre: Skipped 2 previous similar messages [ 1268.745030] Lustre: Skipped 1 previous similar message [ 1271.135728] LustreError: 46617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.183862] Lustre: server umount lustre-MDT0000 complete [ 1273.832392] LustreError: 45019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.853923] LustreError: 45019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1274.752061] LustreError: 45003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394572 with bad export cookie 12253398567019068233 [ 1274.771838] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.777527] LustreError: 45003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1276.253881] LustreError: 47208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.321228] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [ 1276.334153] Lustre: Skipped 1 previous similar message [ 1278.949049] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.988393] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1281.382255] LustreError: 46617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.407735] LustreError: 46617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1284.071482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1284.082548] Lustre: Skipped 2 previous similar messages [ 1289.185164] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1289.499226] Lustre: server umount lustre-MDT0001 complete [ 1304.836193] Lustre: server umount lustre-OST0000 complete [ 1321.622831] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1324.978250] Key type lgssc unregistered [ 1325.332613] LNet: 48884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.343250] LNetError: 48884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.358841] LNet: Removed LNI 192.168.201.113@tcp [ 1326.289188] Key type .llcrypt unregistered [ 1326.291638] Key type ._llcrypt unregistered [ 1353.891792] Key type ._llcrypt registered [ 1353.893390] Key type .llcrypt registered [ 1353.983771] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1355.206730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.236262] alg: No test for adler32 (adler32-zlib) [ 1356.253408] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1356.580847] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1358.293474] Key type lgssc registered [ 1359.318964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1367.493841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1367.506311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.072453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1388.192368] LustreError: 50276:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 1414.880566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.382630] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1428.959064] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1594.123268] Lustre: server umount lustre-OST0000 complete [ 1612.589245] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1615.967113] Key type lgssc unregistered [ 1616.283576] LNet: 51954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.298252] LNetError: 51954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.323943] LNet: Removed LNI 192.168.201.113@tcp [ 1617.220280] Key type .llcrypt unregistered [ 1617.222632] Key type ._llcrypt unregistered [ 1644.110222] Key type ._llcrypt registered [ 1644.112387] Key type .llcrypt registered [ 1644.268561] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1645.961179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.014555] alg: No test for adler32 (adler32-zlib) [ 1647.123299] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1647.483312] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1649.128167] Key type lgssc registered [ 1650.342534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1659.124268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1659.161035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.688117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.337193] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1670.127418] LustreError: 53368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.159086] LustreError: 53368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1675.003394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.235912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1675.433147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1679.791296] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1687.200857] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1692.983972] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1699.533448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1699.784219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.232244] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1707.012210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1715.284253] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1722.338693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1722.345475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.355859] LustreError: Skipped 1 previous similar message [ 1722.363774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.387727] Lustre: Skipped 1 previous similar message [ 1722.411175] Lustre: Skipped 1 previous similar message [ 1724.444894] Lustre: server umount lustre-OST0000 complete [ 1731.558136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1731.565339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.585842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.579602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.997048] Lustre: server umount lustre-MDT0000 complete [ 1737.698495] LustreError: 53365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.726210] LustreError: 53365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1738.791755] LustreError: 55212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395036 with bad export cookie 10047425561933158808 [ 1738.799856] LustreError: 55212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1738.803580] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1739.258746] Lustre: server umount lustre-MDT0001 complete [ 1751.964210] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1755.403986] Key type lgssc unregistered [ 1755.785076] LNet: 56901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.800266] LNetError: 56901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.862327] LNet: Removed LNI 192.168.201.113@tcp [ 1758.468752] Key type .llcrypt unregistered [ 1758.471560] Key type ._llcrypt unregistered [ 1785.541602] Key type ._llcrypt registered [ 1785.544537] Key type .llcrypt registered [ 1785.662938] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1786.871509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.998810] alg: No test for adler32 (adler32-zlib) [ 1788.006836] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 1788.211004] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1789.888242] Key type lgssc registered [ 1791.047442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.324522] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.387354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1814.944735] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1820.065340] LustreError: 58289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 1846.752936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.579788] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1863.362719] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1872.867210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.877280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.004631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1879.022850] LustreError: 58314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1879.059681] LustreError: 58314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1881.069153] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1881.123101] LustreError: 58315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.160656] LustreError: 58315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1886.181604] LustreError: 59326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.201319] LustreError: 59326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1891.306764] LustreError: 59327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.329780] LustreError: 59327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1891.602459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1892.066606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.844158] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1905.312421] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1910.826231] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1914.851471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1914.862328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.885469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1917.922907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1917.929843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1917.946855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.505319] Lustre: server umount lustre-OST0000 complete [ 1943.525718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.531103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1943.557993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.573453] Lustre: Skipped 2 previous similar messages [ 1946.521200] Lustre: server umount lustre-MDT0000 complete [ 1948.644336] LustreError: 59328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.661886] LustreError: 59328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1951.199394] LustreError: 58297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395249 with bad export cookie 17493346404375412594 [ 1951.201971] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.235129] LustreError: 58297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1957.709966] Lustre: server umount lustre-MDT0001 complete [ 1969.314407] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1972.168308] Key type lgssc unregistered [ 1972.451850] LNet: 62035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.456902] LNetError: 62035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.473785] LNet: Removed LNI 192.168.201.113@tcp [ 1973.328583] Key type .llcrypt unregistered [ 1973.330716] Key type ._llcrypt unregistered [ 1997.749631] Key type ._llcrypt registered [ 1997.751709] Key type .llcrypt registered [ 1997.850104] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1998.738116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1998.813324] alg: No test for adler32 (adler32-zlib) [ 1999.961493] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2000.278908] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2001.960170] Key type lgssc registered [ 2003.256593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2010.576407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2010.613240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.080218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.208965] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2022.382712] LustreError: 63449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.410106] LustreError: 63449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2025.613040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.888385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.627206] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2039.351706] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2045.679148] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2053.013706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.361108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.505304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2060.919707] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2069.471858] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2088.417425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2088.423563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.440638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.497526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.508067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.513843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.560033] Lustre: Skipped 1 previous similar message [ 2096.624692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.639723] Lustre: Skipped 1 previous similar message [ 2101.219740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2101.427914] Lustre: server umount lustre-OST0000 complete [ 2106.852092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.867789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2106.887636] Lustre: Skipped 1 previous similar message [ 2106.902535] Lustre: Skipped 1 previous similar message [ 2111.850535] Lustre: server umount lustre-MDT0000 complete [ 2111.980603] LustreError: 63450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.011707] LustreError: 63450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2115.978845] LustreError: 63431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395414 with bad export cookie 13767325803936124059 [ 2115.992525] LustreError: 63431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2116.001254] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2116.504557] Lustre: server umount lustre-MDT0001 complete [ 2128.739114] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2131.755539] Key type lgssc unregistered [ 2131.984933] LNet: 67012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.000670] LNetError: 67012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.033536] LNet: Removed LNI 192.168.201.113@tcp [ 2132.892131] Key type .llcrypt unregistered [ 2132.894068] Key type ._llcrypt unregistered [ 2156.768677] Key type ._llcrypt registered [ 2156.771858] Key type .llcrypt registered [ 2156.915388] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2157.946565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.977400] alg: No test for adler32 (adler32-zlib) [ 2159.010225] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2159.211813] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2160.864169] Key type lgssc registered [ 2161.839676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2170.972473] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:44:26 (1787395466) [ 2177.878539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2177.899122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.407872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.187365] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2189.795151] LustreError: 68712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.828344] LustreError: 68712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2193.212337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.612116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.682514] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2206.572425] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2213.142761] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2219.784904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.050088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.194310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2226.137780] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2234.714508] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2245.187791] Lustre: Failing over lustre-MDT0000 [ 2245.897791] Lustre: server umount lustre-MDT0000 complete [ 2248.086264] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.680194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.710234] Lustre: Skipped 2 previous similar messages [ 2253.158980] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.188555] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2256.319649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.414502] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.565774] Lustre: *** cfs_fail_loc=135, val=0*** [ 2256.699624] LustreError: 71799:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2256.710666] LustreError: 71799:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.113@tcp: cfg command failed: rc = -2 [ 2256.723404] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2256.723404] [ 2256.733924] LustreError: MGC192.168.201.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2256.759718] LustreError: 71787:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2256.772709] LustreError: 71787:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2256.777507] LustreError: 71787:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2256.784554] LustreError: 71787:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2256.805619] Lustre: server umount lustre-MDT0000 complete [ 2256.808855] LustreError: 71787:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2258.274014] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.300847] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2263.397478] LustreError: 70061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.422418] LustreError: 70061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2263.629796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.719337] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.934769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.988837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2264.286935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2268.191926] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2269.176686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2269.198329] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2269.229918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2279.407119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.434641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.436909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.459014] Lustre: Skipped 1 previous similar message [ 2281.509176] Lustre: server umount lustre-OST0000 complete [ 2289.633740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.651370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.652821] LustreError: Skipped 1 previous similar message [ 2289.653743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.682041] Lustre: Skipped 2 previous similar messages [ 2291.528499] Lustre: server umount lustre-MDT0000 complete [ 2294.754661] LustreError: 68707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.768277] LustreError: 68707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2295.172936] LustreError: 68693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395593 with bad export cookie 354127306262128689 [ 2295.179364] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.191181] LustreError: 68693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2295.453180] Lustre: server umount lustre-MDT0001 complete [ 2307.634047] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2310.276717] Key type lgssc unregistered [ 2310.567822] LNet: 73580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.590215] LNetError: 73580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.614932] LNet: Removed LNI 192.168.201.113@tcp [ 2311.302532] Key type .llcrypt unregistered [ 2311.304828] Key type ._llcrypt unregistered [ 2332.592272] Key type ._llcrypt registered [ 2332.594354] Key type .llcrypt registered [ 2332.708408] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2333.417570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.436180] alg: No test for adler32 (adler32-zlib) [ 2334.542824] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2334.728747] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2336.370108] Key type lgssc registered [ 2337.318779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2343.906555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.938687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.222525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.667709] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2355.682754] LustreError: 74992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.700591] LustreError: 74992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2357.275291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.685536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2362.288828] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2370.887676] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2377.084189] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2385.938474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2386.466516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.589378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2393.901656] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2402.851961] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2409.430169] Lustre: Failing over lustre-MDT0000 [ 2409.808205] Lustre: server umount lustre-MDT0000 complete [ 2410.876911] LustreError: 74985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.060583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2414.068965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.075596] LustreError: 74991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.087944] Lustre: Skipped 1 previous similar message [ 2414.124403] LustreError: 74991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2418.555771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.670911] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.797807] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2418.876757] LustreError: 78078:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2418.883527] LustreError: 78078:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.113@tcp: cfg command failed: rc = -9 [ 2418.890905] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2418.890905] [ 2418.900730] LustreError: MGC192.168.201.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2418.911308] LustreError: 78067:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2418.915499] LustreError: 78067:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2418.920143] LustreError: 78067:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2418.923708] LustreError: 78067:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2418.937773] Lustre: server umount lustre-MDT0000 complete [ 2418.939405] LustreError: 78067:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2419.171404] LustreError: 74987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.183585] LustreError: 74987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2424.297168] LustreError: 77240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.323790] LustreError: 77240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2425.546105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.655639] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.663227] LustreError: 74983:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2425.883232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.926784] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2426.208531] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2429.294891] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2430.957518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2431.014145] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2431.060287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2439.649407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.668095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.695308] Lustre: Skipped 1 previous similar message [ 2439.711056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.188848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.190951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.224185] Lustre: Skipped 1 previous similar message [ 2442.824765] Lustre: server umount lustre-OST0000 complete [ 2446.314364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2446.315950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.317212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.338773] Lustre: Skipped 1 previous similar message [ 2451.426045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.438419] Lustre: Skipped 1 previous similar message [ 2452.023341] Lustre: server umount lustre-MDT0000 complete [ 2455.447635] LustreError: 74971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395753 with bad export cookie 2687195411022792830 [ 2455.455079] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.465855] LustreError: 74971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2455.726550] Lustre: server umount lustre-MDT0001 complete [ 2465.436369] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2467.961955] Key type lgssc unregistered [ 2468.197513] LNet: 79859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.203805] LNetError: 79859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.222685] LNet: Removed LNI 192.168.201.113@tcp [ 2468.972189] Key type .llcrypt unregistered [ 2468.974214] Key type ._llcrypt unregistered [ 2479.564520] Key type ._llcrypt registered [ 2479.570395] Key type .llcrypt registered [ 2479.664325] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2479.935513] Key type .llcrypt unregistered [ 2479.938325] Key type ._llcrypt unregistered [ 2500.034273] Key type ._llcrypt registered [ 2500.037592] Key type .llcrypt registered [ 2500.133391] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2500.777586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.818419] alg: No test for adler32 (adler32-zlib) [ 2501.858456] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2502.092509] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2503.729311] Key type lgssc registered [ 2504.487668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2510.739484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2510.777224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.291810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.083910] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2522.599108] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.618627] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2523.050622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.283980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2526.674810] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2532.986531] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2537.857465] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2543.955887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2544.286942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.816793] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2556.189856] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2557.627266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2568.162454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2568.165667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.168144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.172074] Lustre: Skipped 1 previous similar message [ 2573.289804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.310453] Lustre: Skipped 2 previous similar messages [ 2573.601520] Lustre: server umount lustre-OST0000 complete [ 2578.402078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.406497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.422390] Lustre: Skipped 1 previous similar message [ 2583.358742] Lustre: server umount lustre-MDT0000 complete [ 2583.524981] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.539436] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2587.274790] LustreError: 81745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395885 with bad export cookie 4255286765096953237 [ 2587.283958] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.658418] Lustre: server umount lustre-MDT0001 complete [ 2600.499768] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2603.305159] Key type lgssc unregistered [ 2603.647811] LNet: 85294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.662165] LNetError: 85294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.676523] LNet: Removed LNI 192.168.201.113@tcp [ 2604.713742] Key type .llcrypt unregistered [ 2604.718053] Key type ._llcrypt unregistered [ 2625.788832] Key type ._llcrypt registered [ 2625.792709] Key type .llcrypt registered [ 2625.940810] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2627.340837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2627.412079] alg: No test for adler32 (adler32-zlib) [ 2628.463939] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2628.653627] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2630.289627] Key type lgssc registered [ 2631.709560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.146191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.179288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.712309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.405470] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2653.178791] LustreError: 86699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2653.211209] LustreError: 86699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2658.274721] LustreError: 86698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.617131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.931063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2663.821964] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2673.216505] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2679.603239] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2687.477467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2687.866236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.906741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2693.624916] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2700.541394] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2713.574174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.591957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2713.606460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2713.607875] Lustre: Skipped 1 previous similar message [ 2717.141058] Lustre: server umount lustre-OST0000 complete [ 2723.812309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.814081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2723.845120] Lustre: Skipped 1 previous similar message [ 2723.878608] Lustre: Skipped 2 previous similar messages [ 2727.713800] Lustre: server umount lustre-MDT0000 complete [ 2728.931795] LustreError: 87702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.961883] LustreError: 87702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2731.223367] LustreError: 86680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396029 with bad export cookie 9626030171165105051 [ 2731.230801] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.232888] LustreError: 86680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2731.507096] Lustre: server umount lustre-MDT0001 complete [ 2741.373774] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2744.044777] Key type lgssc unregistered [ 2744.295278] LNet: 90229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2744.311713] LNetError: 90229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2744.341427] LNet: Removed LNI 192.168.201.113@tcp [ 2745.078173] Key type .llcrypt unregistered [ 2745.079523] Key type ._llcrypt unregistered [ 2766.362597] Key type ._llcrypt registered [ 2766.365654] Key type .llcrypt registered [ 2766.452907] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2767.305546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2767.316407] alg: No test for adler32 (adler32-zlib) [ 2768.374187] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2768.579256] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2770.240143] Key type lgssc registered [ 2771.072854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2777.229742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2777.252066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.587869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.491847] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2788.844466] LustreError: 91644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.854891] LustreError: 91644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2791.954330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.385137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2797.263213] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2804.149764] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2809.906914] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2816.101499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.447363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.559523] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2823.607494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2829.544973] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2843.107803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2843.111392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.115311] LustreError: Skipped 1 previous similar message [ 2843.118413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.140051] Lustre: Skipped 1 previous similar message [ 2844.137202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.149502] Lustre: Skipped 1 previous similar message [ 2848.130232] Lustre: server umount lustre-OST0000 complete [ 2853.857774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.867290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.883363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.894688] Lustre: Skipped 1 previous similar message [ 2858.318448] Lustre: server umount lustre-MDT0000 complete [ 2859.506532] LustreError: 91643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.554620] LustreError: 91643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2861.857076] LustreError: 91625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396159 with bad export cookie 7505959911379631402 [ 2861.859324] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.868774] LustreError: 91625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2862.353433] Lustre: server umount lustre-MDT0001 complete [ 2874.049782] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2876.841342] Key type lgssc unregistered [ 2877.126465] LNet: 95197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2877.131378] LNetError: 95197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2877.152831] LNet: Removed LNI 192.168.201.113@tcp [ 2877.736984] Key type .llcrypt unregistered [ 2877.738895] Key type ._llcrypt unregistered [ 2902.575754] Key type ._llcrypt registered [ 2902.581573] Key type .llcrypt registered [ 2902.703599] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2903.674098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2903.690958] alg: No test for adler32 (adler32-zlib) [ 2904.864906] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 2905.125902] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2906.800563] Key type lgssc registered [ 2907.950651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.776791] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.798085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.296396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2936.416328] LustreError: 96583:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 2963.104216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.605393] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2977.372689] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2982.152699] Lustre: server umount lustre-OST0000 complete [ 2990.251228] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:58:06 (1787396286) [ 2997.239782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.748207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.207434] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3002.853030] LustreError: 98221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.887688] LustreError: 98221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3007.982098] LustreError: 98222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.516972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.763299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3016.048431] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3022.994484] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3028.057503] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3035.120137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.634988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.551946] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3043.849399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3049.712266] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3064.293761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.297397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3064.317153] Lustre: Skipped 1 previous similar message [ 3064.341943] Lustre: Skipped 1 previous similar message [ 3069.410095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.427450] Lustre: Skipped 1 previous similar message [ 3070.670559] Lustre: server umount lustre-OST0000 complete [ 3078.628491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3078.636418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.650779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.649615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.223328] Lustre: server umount lustre-MDT0000 complete [ 3084.774597] LustreError: 98216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.793304] LustreError: 98216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3086.516513] LustreError: 98204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396384 with bad export cookie 7308515844332995467 [ 3086.529579] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.070385] Lustre: server umount lustre-MDT0001 complete [ 3099.491936] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3102.153376] Key type lgssc unregistered [ 3102.457586] LNet: 101913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3102.467989] LNetError: 101913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3102.492174] LNet: Removed LNI 192.168.201.113@tcp [ 3103.394647] Key type .llcrypt unregistered [ 3103.401706] Key type ._llcrypt unregistered [ 3148.118278] Key type ._llcrypt registered [ 3148.121225] Key type .llcrypt registered [ 3148.320976] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3149.831808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.988084] alg: No test for adler32 (adler32-zlib) [ 3151.490093] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3151.833129] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3153.648176] Key type lgssc registered [ 3155.577600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3166.100957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3166.138813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.748166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.024670] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3177.962080] LustreError: 104115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.982663] LustreError: 104115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3183.084722] LustreError: 104114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.206088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.553937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3190.117199] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3199.766670] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3209.733560] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3219.114239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.465855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.382508] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3233.485236] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3237.982783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3248.101417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3248.105961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.121799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3248.157976] Lustre: Skipped 1 previous similar message [ 3253.221444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3253.232225] Lustre: Skipped 1 previous similar message [ 3253.674111] Lustre: server umount lustre-OST0000 complete [ 3258.340311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.354144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3258.363497] Lustre: Skipped 1 previous similar message [ 3258.385211] Lustre: Skipped 2 previous similar messages [ 3263.459034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.479385] Lustre: Skipped 1 previous similar message [ 3265.684464] Lustre: server umount lustre-MDT0000 complete [ 3268.578333] LustreError: 104109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.595898] LustreError: 104109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3269.511825] LustreError: 104095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396567 with bad export cookie 3175790022721303814 [ 3269.525202] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3269.531162] LustreError: 104095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3270.045601] Lustre: server umount lustre-MDT0001 complete [ 3280.225501] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3282.566861] Key type lgssc unregistered [ 3282.781339] LNet: 107670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3282.788484] LNetError: 107670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3282.808041] LNet: Removed LNI 192.168.201.113@tcp [ 3283.548581] Key type .llcrypt unregistered [ 3283.550703] Key type ._llcrypt unregistered [ 3301.853746] Key type ._llcrypt registered [ 3301.857415] Key type .llcrypt registered [ 3301.966950] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3302.981161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.048703] alg: No test for adler32 (adler32-zlib) [ 3304.168890] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3304.387638] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3306.096184] Key type lgssc registered [ 3307.084394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.854370] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3314.868300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.464933] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3335.585989] LustreError: 108913:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 3362.272206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.133638] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3377.664561] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3386.196044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.496774] LustreError: MGC192.168.201.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3386.505214] LustreError: 109937:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3386.510608] LustreError: 109937:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3386.522029] LustreError: 109937:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3386.591925] Lustre: server umount lustre-MDT0000 complete [ 3386.597583] LustreError: 109937:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3405.280175] Lustre: 108250:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396687/real 1787396687] req@00000000b2f34877 x1874221179342208/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787396703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3405.319444] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.326518] LustreError: 108925:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3425.888576] Lustre: server umount lustre-OST0000 complete [ 3437.101546] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 3448.286821] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3490.804845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.083658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3491.100567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3491.150773] Lustre: lustre-MDT0000: new disk, initializing [ 3491.209560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.237784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3496.662704] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3508.975562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.128029] Lustre: 112738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3509.250729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3509.257027] Lustre: Skipped 1 previous similar message [ 3509.355303] Lustre: lustre-MDT0001: new disk, initializing [ 3509.476177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.539715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3509.548783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3514.408889] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3524.822215] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3531.706533] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3539.175546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3539.420790] Lustre: lustre-OST0000: new disk, initializing [ 3539.425472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3539.430210] Lustre: 114710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3539.489214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.048749] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3549.768909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3549.783812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3549.932114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3556.092313] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3560.904465] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3561.108849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3565.312417] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3565.473767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3570.677813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.681758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.683219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.695860] Lustre: Skipped 1 previous similar message [ 3573.449513] Lustre: server umount lustre-OST0000 complete [ 3580.905110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.906865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.916869] Lustre: Skipped 1 previous similar message [ 3580.921767] Lustre: Skipped 1 previous similar message [ 3584.713782] Lustre: server umount lustre-MDT0000 complete [ 3588.447240] LustreError: 112732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396886 with bad export cookie 7522410534732430856 [ 3588.452909] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3588.454636] LustreError: 112732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3588.887400] Lustre: server umount lustre-MDT0001 complete [ 3599.070385] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:08:14 (1787396894) [ 3602.887662] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3622.155571] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 3632.726727] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3671.849636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.269294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3672.313434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3672.479542] Lustre: lustre-MDT0000: new disk, initializing [ 3672.651842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.687321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3677.667814] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3689.966826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.159038] Lustre: 119609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3690.324614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3690.328984] Lustre: Skipped 1 previous similar message [ 3690.468283] Lustre: lustre-MDT0001: new disk, initializing [ 3690.614783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.668033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3690.679897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3695.027140] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3706.412621] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3713.640728] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3719.893589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3720.207290] Lustre: lustre-OST0000: new disk, initializing [ 3720.211889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3720.218892] Lustre: 121577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3720.315156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.836606] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3727.442468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3727.458678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3727.610103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3738.200510] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3745.543063] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3746.069958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3753.557386] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3753.855892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3758.571635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3758.574446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.575129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.575134] Lustre: Skipped 1 previous similar message [ 3758.595137] LustreError: Skipped 1 previous similar message [ 3758.633308] Lustre: Skipped 1 previous similar message [ 3762.922464] Lustre: server umount lustre-OST0000 complete [ 3767.266935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3767.282561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.300260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.310772] Lustre: Skipped 1 previous similar message [ 3773.035420] Lustre: server umount lustre-MDT0000 complete [ 3773.933721] LustreError: 119622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3773.959774] LustreError: 119622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3777.347674] LustreError: 120414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397075 with bad export cookie 7522410534732431731 [ 3777.360268] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.362372] LustreError: 120414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3777.692145] Lustre: server umount lustre-MDT0001 complete [ 3787.374252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3787.879397] LustreError: 123796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.029504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3792.693497] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3793.380513] LustreError: 123797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.400757] LustreError: 123797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3797.490499] LustreError: 123796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.606572] LustreError: 123797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.536158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.732823] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3818.443845] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3825.518186] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3832.707140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3840.382307] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3848.812888] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3866.635928] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3873.257237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.258948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.276315] Lustre: Skipped 1 previous similar message [ 3873.286734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3873.306356] Lustre: Skipped 3 previous similar messages [ 3878.719857] Lustre: server umount lustre-OST0000 complete [ 3883.497903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.512015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3883.512670] Lustre: Skipped 1 previous similar message [ 3883.548129] Lustre: Skipped 3 previous similar messages [ 3889.673484] Lustre: server umount lustre-MDT0000 complete [ 3893.465483] LustreError: 123778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397191 with bad export cookie 7522410534732432620 [ 3893.476550] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3893.821392] Lustre: server umount lustre-MDT0001 complete [ 3906.128912] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3909.372348] Key type lgssc unregistered [ 3909.663532] LNet: 127449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3909.671469] LNetError: 127449:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.685860] LNet: Removed LNI 192.168.201.113@tcp [ 3910.631963] Key type .llcrypt unregistered [ 3910.634361] Key type ._llcrypt unregistered [ 3930.925723] Key type ._llcrypt registered [ 3930.928521] Key type .llcrypt registered [ 3931.020081] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 3946.573148] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3947.621911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3947.787799] alg: No test for adler32 (adler32-zlib) [ 3948.922893] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 3949.154062] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3950.848182] Key type lgssc registered [ 3951.746489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3994.288518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3994.327846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3995.567665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3995.587646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3995.642721] Lustre: lustre-MDT0000: new disk, initializing [ 3995.700891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.715489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4000.441698] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4011.968358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.251666] Lustre: 130605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4012.344782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4012.351265] Lustre: Skipped 1 previous similar message [ 4012.445929] Lustre: lustre-MDT0001: new disk, initializing [ 4012.593771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4012.662514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4012.692688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4018.001426] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4028.475741] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4035.016941] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4042.829609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4043.207510] Lustre: lustre-OST0000: new disk, initializing [ 4043.212966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4043.218901] Lustre: 132573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4043.382326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4049.525114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4049.540328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4049.747128] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4050.744583] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4061.324128] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4067.489956] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4067.785909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4072.958302] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4073.200278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4075.489912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4075.490711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4075.504864] LustreError: Skipped 1 previous similar message [ 4075.506272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4075.548763] Lustre: Skipped 1 previous similar message [ 4080.612866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4080.627520] Lustre: Skipped 1 previous similar message [ 4081.516178] Lustre: server umount lustre-OST0000 complete [ 4085.730482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.742316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4085.763682] Lustre: Skipped 1 previous similar message [ 4085.775630] Lustre: Skipped 2 previous similar messages [ 4090.862621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.874877] Lustre: Skipped 1 previous similar message [ 4091.643747] Lustre: server umount lustre-MDT0000 complete [ 4095.973591] LustreError: 130615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.025363] LustreError: 130615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4097.616161] LustreError: 130596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397395 with bad export cookie 4956226763169570655 [ 4097.624800] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.626941] LustreError: 130596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4098.144607] Lustre: server umount lustre-MDT0001 complete [ 4110.342215] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:16:45 (1787397405) [ 4119.573799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4120.233760] LustreError: 134985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.300161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.349672] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4125.666252] LustreError: 134986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.787397] LustreError: 134985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.851351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.145842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.285633] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4144.663171] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4150.562107] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4153.828484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4153.837245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.857552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4155.875402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4158.851690] Lustre: server umount lustre-MDT0000 complete [ 4158.953244] LustreError: 134986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.943410] LustreError: 134967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397460 with bad export cookie 4956226763169571537 [ 4162.948136] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.951434] LustreError: 134967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4163.359535] Lustre: server umount lustre-MDT0001 complete [ 4173.281480] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:17:48 (1787397468) [ 4180.892211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.385251] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4201.504518] LustreError: 137401:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 4227.299598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.300238] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4241.937835] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4244.570071] Lustre: server umount lustre-OST0000 complete [ 4254.415953] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:19:10 (1787397550) [ 4261.776072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.530920] LustreError: 138848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.551145] LustreError: 138848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4262.678690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.482302] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4278.060689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.480449] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4291.349604] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4297.209918] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4304.992489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4305.475326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.482890] Lustre: Skipped 1 previous similar message [ 4312.627121] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4321.835650] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4339.176758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.181521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.192413] Lustre: Skipped 2 previous similar messages [ 4339.195761] Lustre: Skipped 3 previous similar messages [ 4341.799584] Lustre: server umount lustre-MDT0000 complete [ 4344.298786] LustreError: 139852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.322323] LustreError: 139852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4346.359131] LustreError: 138830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397644 with bad export cookie 4956226763169572202 [ 4346.361784] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.372608] LustreError: 138830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4346.786745] Lustre: server umount lustre-MDT0001 complete [ 4361.528122] Lustre: server umount lustre-OST0000 complete [ 4370.539689] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:21:06 (1787397666) [ 4376.856449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.418626] LustreError: 142526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.535158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.470484] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4392.027184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4396.953904] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4405.544443] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4413.336955] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4421.009755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4422.397459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4429.162714] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4437.893296] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4443.107255] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4443.310805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4448.476988] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4448.729726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4453.348677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.354622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4453.395322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.416605] Lustre: Skipped 2 previous similar messages [ 4456.716797] Lustre: server umount lustre-OST0000 complete [ 4466.565924] Lustre: server umount lustre-MDT0000 complete [ 4468.708535] LustreError: 142527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.734875] LustreError: 142527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4470.945696] LustreError: 142507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397768 with bad export cookie 4956226763169573266 [ 4470.953170] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4470.957843] LustreError: 142507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4471.393807] Lustre: server umount lustre-MDT0001 complete [ 4481.121686] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:22:56 (1787397776) [ 4488.404452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4503.969154] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4509.088474] LustreError: 146686:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 4534.753997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.772374] Lustre: Skipped 2 previous similar messages [ 4541.627771] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4549.068360] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4556.111053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.933561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4561.730896] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4571.655774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.865498] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4584.986605] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4591.203573] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4597.032096] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4597.369359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4603.604443] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4603.901570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4607.970774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4607.981461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.001456] Lustre: Skipped 4 previous similar messages [ 4608.001837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4608.029560] Lustre: Skipped 3 previous similar messages [ 4612.637168] Lustre: server umount lustre-MDT0000 complete [ 4613.091384] LustreError: 146711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.134171] LustreError: 146711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4616.585756] LustreError: 146693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397914 with bad export cookie 4956226763169574295 [ 4616.592964] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.603386] LustreError: 146693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4617.008228] Lustre: server umount lustre-MDT0001 complete [ 4630.248537] Lustre: server umount lustre-OST0000 complete [ 4640.120751] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:25:35 (1787397935) [ 4647.406905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.072893] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4668.192507] LustreError: 150882:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 4700.223459] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4709.198701] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4715.691665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4716.166140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4720.888338] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4729.097273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.755595] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4742.243886] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4747.732068] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4754.474641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.668253] Lustre: lustre-OST0001: new disk, initializing [ 4754.671348] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4754.675932] Lustre: 153755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4760.687661] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4764.787164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4764.800289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4764.939832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4769.465911] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4773.653509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4773.837470] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4778.782593] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4779.026765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4782.562096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4782.577416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.595926] Lustre: Skipped 1 previous similar message [ 4782.604077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.360281] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4795.489370] Lustre: server umount lustre-OST0000 complete [ 4800.998287] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4801.008972] LustreError: Skipped 1 previous similar message [ 4813.280172] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4813.435435] Lustre: server umount lustre-OST0001 complete [ 4823.659422] Lustre: server umount lustre-MDT0000 complete [ 4826.840438] LustreError: 150889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398124 with bad export cookie 4956226763169575247 [ 4826.848281] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4826.851646] LustreError: 150889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4827.133788] Lustre: server umount lustre-MDT0001 complete [ 4851.199668] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:29:07 (1787398147) [ 4852.819323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4854.298995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4856.020380] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:29:12 (1787398152) [ 4862.228851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.678777] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4862.712043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4862.850359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.855673] Lustre: Skipped 6 previous similar messages [ 4867.185723] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4875.173911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.252832] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4875.285201] Lustre: 157325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4879.432244] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4886.379747] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4891.973438] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4898.769455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4899.109865] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4906.110576] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4908.324825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4914.965487] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4919.812747] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4920.072886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4925.100458] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4925.291418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4928.998197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4929.000407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4929.016061] Lustre: Skipped 5 previous similar messages [ 4929.019252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4929.019260] Lustre: Skipped 12 previous similar messages [ 4933.558355] Lustre: server umount lustre-OST0000 complete [ 4956.447362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.860549] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4969.077083] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.090139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.090646] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4969.113066] Lustre: Skipped 1 previous similar message [ 4972.735701] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.249894] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4978.470843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4983.638620] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4983.890389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4990.177574] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9daf5089e000.ost_server_uuid 50 [ 4992.197596] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9daf5089e000.ost_server_uuid in FULL state after 0 sec [ 4992.379808] LustreError: 157333:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.13@tcp) returned error from blocking AST (req@00000000ccac3859 x1874221855939584 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000001e19bdc6/0x44c80dfb5d01baa9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.13@tcp remote: 0x2e9224d849561dd6 expref: 5 pid: 157332 timeout: 5092 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4992.398094] LustreError: lustre-MDT0000: A client on nid 192.168.201.13@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4992.404117] LustreError: 157321:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.13@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000001e19bdc6/0x44c80dfb5d01baa9 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.13@tcp remote: 0x2e9224d849561dd6 expref: 6 pid: 157332 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4998.118462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4998.135847] LustreError: Skipped 1 previous similar message [ 5003.034353] Lustre: server umount lustre-OST0000 complete [ 5013.282158] Lustre: server umount lustre-MDT0000 complete [ 5015.522039] LustreError: 162400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.537923] LustreError: 162400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 5017.077055] LustreError: 157318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398315 with bad export cookie 4956226763169576430 [ 5017.080412] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.089847] LustreError: 157318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5031.392191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5031.687736] Lustre: server umount lustre-MDT0001 complete [ 5043.347979] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5046.306448] Key type lgssc unregistered [ 5046.628173] LNet: 163441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5046.643349] LNetError: 163441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5047.722376] LNet: Removed LNI 192.168.201.113@tcp [ 5048.818185] Key type .llcrypt unregistered [ 5048.820081] Key type ._llcrypt unregistered [ 5075.953289] Key type ._llcrypt registered [ 5075.954824] Key type .llcrypt registered [ 5076.188643] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5077.481559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5077.707548] alg: No test for adler32 (adler32-zlib) [ 5078.817629] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5079.097917] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5080.808438] Key type lgssc registered [ 5082.959697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5091.639395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5091.655326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.091875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.576653] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5103.588543] LustreError: 164857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.604415] LustreError: 164857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5106.670539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.148471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5112.501401] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5120.213647] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5126.992524] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5133.300523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5133.753378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.888250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5140.402526] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5147.891580] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5156.694594] Lustre: Failing over lustre-MDT0000 [ 5156.864137] Lustre: server umount lustre-MDT0000 complete [ 5157.367790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.369520] LustreError: 164854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.384017] Lustre: Skipped 2 previous similar messages [ 5157.419809] LustreError: 164854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5162.465839] LustreError: 165585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.482697] LustreError: 165585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5167.585197] LustreError: 164857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.598941] LustreError: 164857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5169.139468] LustreError: 164839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.13@tcp arrived at 1787398467 with bad export cookie 5819038492837392352 [ 5169.151357] LustreError: 164839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5172.707869] LustreError: 165826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.757650] LustreError: 165826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5173.536500] Lustre: 164168:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398455/real 1787398455] req@0000000038ae05c0 x1874223040588032/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787398471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5173.571700] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5176.763460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.969385] LustreError: 164857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.992961] LustreError: 164857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5184.275778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.324273] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5189.331820] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5189.611146] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5189.615184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5191.598186] Lustre: lustre-MDT0000: Denying connection for new client a886d7a6-9ce6-4064-b25f-ffb4590410a4 (at 192.168.201.13@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5199.847313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.856141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.865120] Lustre: Skipped 1 previous similar message [ 5204.967556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.981182] Lustre: Skipped 1 previous similar message [ 5210.089168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5210.098428] Lustre: Skipped 1 previous similar message [ 5214.176696] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5214.338622] Lustre: server umount lustre-OST0000 complete [ 5218.650393] LustreError: 168591:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5218.656132] Lustre: 167938:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5218.658988] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5218.674899] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5218.682268] LustreError: 167938:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002956b3fc x1874223040609792/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5218.695281] LustreError: 167938:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5218.703416] LustreError: 167938:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5218.715896] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5219.116624] Lustre: server umount lustre-MDT0000 complete [ 5220.322547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.337857] Lustre: Skipped 1 previous similar message [ 5220.350854] LustreError: 164853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.313765] LustreError: 167435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398521 with bad export cookie 5819038492837392457 [ 5223.322471] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.327340] LustreError: 167435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5226.055752] Lustre: server umount lustre-MDT0001 complete [ 5240.757930] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5243.192503] Key type lgssc unregistered [ 5243.424924] LNet: 169345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5243.430033] LNetError: 169345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5243.443871] LNet: Removed LNI 192.168.201.113@tcp [ 5244.420258] Key type .llcrypt unregistered [ 5244.422969] Key type ._llcrypt unregistered [ 5268.734321] Key type ._llcrypt registered [ 5268.738680] Key type .llcrypt registered [ 5268.909790] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5270.497748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5270.522079] alg: No test for adler32 (adler32-zlib) [ 5271.597450] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5271.908944] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5273.552230] Key type lgssc registered [ 5274.965372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5282.549786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5282.613532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.362975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5289.014915] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5294.561450] LustreError: 170756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.583438] LustreError: 170756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5298.493693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.858310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5303.355288] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5312.189992] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5318.932663] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5327.487305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5327.848782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.920386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5334.777580] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5343.703576] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5353.441502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5353.445293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.449649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5353.476199] Lustre: Skipped 1 previous similar message [ 5358.562654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.572433] Lustre: Skipped 2 previous similar messages [ 5363.682140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.699148] Lustre: Skipped 1 previous similar message [ 5365.220033] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5365.457395] Lustre: server umount lustre-OST0000 complete [ 5370.337637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5370.353527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.368485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.922439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.502941] Lustre: server umount lustre-MDT0000 complete [ 5379.043809] LustreError: 171484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.073822] LustreError: 171484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5380.820996] LustreError: 170735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398678 with bad export cookie 10507369103573046190 [ 5380.830255] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5381.021279] Lustre: server umount lustre-MDT0001 complete [ 5395.315669] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5397.956754] Key type lgssc unregistered [ 5398.179555] LNet: 174289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5398.185614] LNetError: 174289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5399.210180] LNet: Removed LNI 192.168.201.113@tcp [ 5400.112351] Key type .llcrypt unregistered [ 5400.113858] Key type ._llcrypt unregistered [ 5434.653461] Key type ._llcrypt registered [ 5434.656813] Key type .llcrypt registered [ 5434.780623] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5436.444036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5436.507162] alg: No test for adler32 (adler32-zlib) [ 5438.101954] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5438.537113] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5440.424263] Key type lgssc registered [ 5442.014994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5452.211893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5452.256926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.830843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5459.199552] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5463.016975] LustreError: 175996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.055620] LustreError: 175996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5468.132159] LustreError: 175997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.127563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.835023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5476.698785] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5486.158723] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5492.562536] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5499.829565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5500.165376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5507.577810] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5516.731711] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5523.710877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5536.226047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5536.237972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.253960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5539.299963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5539.304755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.965778] Lustre: server umount lustre-OST0000 complete [ 5547.492930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5547.514546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.548901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5547.557408] Lustre: Skipped 1 previous similar message [ 5549.540043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.561926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5549.566067] Lustre: Skipped 1 previous similar message [ 5551.591662] Lustre: server umount lustre-MDT0000 complete [ 5554.657242] LustreError: 175996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.687544] LustreError: 175996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5556.110518] LustreError: 176732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398854 with bad export cookie 1950159257534840485 [ 5556.112348] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5556.134064] LustreError: 176732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5556.527702] Lustre: server umount lustre-MDT0001 complete [ 5574.851699] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5577.849925] Key type lgssc unregistered [ 5578.246968] LNet: 179575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5578.262256] LNetError: 179575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5578.280915] LNet: Removed LNI 192.168.201.113@tcp [ 5579.184462] Key type .llcrypt unregistered [ 5579.198475] Key type ._llcrypt unregistered [ 5603.979468] Key type ._llcrypt registered [ 5603.982668] Key type .llcrypt registered [ 5604.081830] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5605.201242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5605.344394] alg: No test for adler32 (adler32-zlib) [ 5606.483464] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5606.720369] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5608.440195] Key type lgssc registered [ 5609.912689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5623.089038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5623.136273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.687369] Lustre: *** cfs_fail_loc=135, val=0*** [ 5624.823495] LustreError: 181101:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5624.843580] LustreError: 181101:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.113@tcp: cfg command failed: rc = -2 [ 5624.873916] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5624.873916] [ 5624.907184] LustreError: MGC192.168.201.113@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5624.953775] LustreError: 181062:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5624.976123] LustreError: 181062:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5624.991600] LustreError: 181062:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5625.008916] LustreError: 181062:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5625.089824] Lustre: server umount lustre-MDT0000 complete [ 5625.094214] LustreError: 181062:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5634.404975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.295832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5640.588854] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5645.794285] LustreError: 181473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.815751] LustreError: 181473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5650.446678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.064060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5655.551916] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5664.122439] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5669.960991] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5672.662687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.666741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5672.688154] Lustre: Skipped 1 previous similar message [ 5672.712773] Lustre: Skipped 1 previous similar message [ 5675.810066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5675.820981] Lustre: Skipped 1 previous similar message [ 5679.035688] Lustre: server umount lustre-MDT0000 complete [ 5681.122454] LustreError: 181473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.141150] LustreError: 181473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5683.404103] LustreError: 182210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398981 with bad export cookie 17529676756163472129 [ 5683.412997] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5683.427787] LustreError: 182210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5684.007662] Lustre: server umount lustre-MDT0001 complete [ 5695.698286] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5698.265940] Key type lgssc unregistered [ 5698.531479] LNet: 183794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5698.545492] LNetError: 183794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5698.558347] LNet: Removed LNI 192.168.201.113@tcp [ 5699.489444] Key type .llcrypt unregistered [ 5699.491913] Key type ._llcrypt unregistered [ 5723.130349] Key type ._llcrypt registered [ 5723.133912] Key type .llcrypt registered [ 5723.321200] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5723.685660] Key type .llcrypt unregistered [ 5723.689293] Key type ._llcrypt unregistered [ 5745.116890] Key type ._llcrypt registered [ 5745.120357] Key type .llcrypt registered [ 5745.256887] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5746.477552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5746.559457] alg: No test for adler32 (adler32-zlib) [ 5748.039635] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5748.384685] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5750.152182] Key type lgssc registered [ 5751.737653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5760.863539] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5760.875047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.418100] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5781.536599] LustreError: 185929:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 5808.288235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.778772] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5823.065189] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5831.524427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.917921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.981427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5837.043711] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5837.287516] LustreError: 185954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.294718] LustreError: 185954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5842.401695] LustreError: 185955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.431301] LustreError: 185954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.484110] LustreError: 185954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5847.451652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.976476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5853.388403] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5864.090980] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5873.212305] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5878.396867] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5893.604035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5893.609667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.628382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.126206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.130857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.161865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.161875] Lustre: Skipped 1 previous similar message [ 5899.248043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5899.258521] Lustre: Skipped 1 previous similar message [ 5904.394973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5904.397903] Lustre: Skipped 1 previous similar message [ 5905.376379] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5905.645870] Lustre: server umount lustre-OST0000 complete [ 5914.594728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5914.595616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.657156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5917.170338] Lustre: server umount lustre-MDT0000 complete [ 5919.715505] LustreError: 186967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.731579] LustreError: 186967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5921.851105] LustreError: 185935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399219 with bad export cookie 14612751275265066638 [ 5921.856571] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5921.878382] LustreError: 185935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5922.277976] Lustre: server umount lustre-MDT0001 complete [ 5937.658995] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5940.471200] Key type lgssc unregistered [ 5940.764166] LNet: 189898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5940.814905] LNetError: 189898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5941.865313] LNet: Removed LNI 192.168.201.113@tcp [ 5942.759374] Key type .llcrypt unregistered [ 5942.762800] Key type ._llcrypt unregistered [ 5969.937983] Key type ._llcrypt registered [ 5969.939968] Key type .llcrypt registered [ 5970.044268] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5971.248972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5971.380296] alg: No test for adler32 (adler32-zlib) [ 5972.547588] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 5972.790297] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5974.488722] Key type lgssc registered [ 5975.738263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5984.355627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5984.375809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.800455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.318342] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5994.980185] LustreError: 191315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.003844] LustreError: 191315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6000.115799] LustreError: 191316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.336625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.932782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.088744] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6016.233221] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6022.734842] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6030.153920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6030.644339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6031.693654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6038.506391] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6046.650585] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6051.988792] Lustre: Failing over lustre-MDT0000 [ 6052.326595] Lustre: server umount lustre-MDT0000 complete [ 6052.834296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6052.845632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.863375] LustreError: 191316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.425482] LustreError: 191315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.433430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.447682] LustreError: 191315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6056.467431] Lustre: Skipped 1 previous similar message [ 6061.538485] LustreError: 192045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.573485] LustreError: 192045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6069.599588] LustreError: 193812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.627993] LustreError: 193812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6071.726067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.833344] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6072.058868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6072.102952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6074.724935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6076.871908] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6077.437111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6077.468545] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6077.512587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6081.974929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6089.360342] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6103.021893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.040407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.045533] Lustre: Skipped 1 previous similar message [ 6108.132324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6108.148680] Lustre: Skipped 1 previous similar message [ 6108.767712] Lustre: server umount lustre-OST0000 complete [ 6113.249364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6113.251554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.257741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6113.263617] Lustre: Skipped 2 previous similar messages [ 6113.287072] Lustre: Skipped 1 previous similar message [ 6118.370058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6118.379281] Lustre: Skipped 1 previous similar message [ 6118.768093] Lustre: server umount lustre-MDT0000 complete [ 6122.456263] LustreError: 191296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399420 with bad export cookie 12568291153462558377 [ 6122.468231] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6122.816925] Lustre: server umount lustre-MDT0001 complete [ 6133.689992] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6135.662573] Key type lgssc unregistered [ 6135.855798] LNet: 196226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6135.861751] LNetError: 196226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6136.937176] LNet: Removed LNI 192.168.201.113@tcp [ 6137.590568] Key type .llcrypt unregistered [ 6137.607783] Key type ._llcrypt unregistered [ 6158.831267] Key type ._llcrypt registered [ 6158.833882] Key type .llcrypt registered [ 6158.950138] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6159.958106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6159.997458] alg: No test for adler32 (adler32-zlib) [ 6160.958767] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6161.157850] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6162.802240] Key type lgssc registered [ 6163.873210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6170.775640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6170.798894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.268603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.374771] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6182.371848] LustreError: 197641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.396846] LustreError: 197641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6186.797698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.219743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6193.146818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6201.893759] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6207.392600] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6215.113327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6215.654697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6217.737418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6223.129934] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6230.701249] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6234.641771] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6242.281509] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6255.782569] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6262.753491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6262.758742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.768013] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6262.782654] Lustre: Skipped 1 previous similar message [ 6267.377323] Lustre: server umount lustre-OST0000 complete [ 6272.994388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.996912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6273.005579] Lustre: Skipped 1 previous similar message [ 6273.019575] Lustre: Skipped 1 previous similar message [ 6277.663791] Lustre: server umount lustre-MDT0000 complete [ 6278.115769] LustreError: 197641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.132694] LustreError: 197641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6282.212540] LustreError: 198370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.226334] LustreError: 198370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6282.282697] LustreError: 197620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399580 with bad export cookie 13513189339170657375 [ 6282.297082] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.510118] Lustre: server umount lustre-MDT0001 complete [ 6295.596448] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6299.336321] Key type lgssc unregistered [ 6299.564866] LNet: 201495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6299.575251] LNetError: 201495:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6299.593931] LNet: Removed LNI 192.168.201.113@tcp [ 6300.515416] Key type .llcrypt unregistered [ 6300.517119] Key type ._llcrypt unregistered [ 6320.003540] Key type ._llcrypt registered [ 6320.006372] Key type .llcrypt registered [ 6320.098536] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6321.002656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6321.133015] alg: No test for adler32 (adler32-zlib) [ 6322.280264] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6322.554773] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6324.232312] Key type lgssc registered [ 6325.257722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6332.457788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6332.474849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.881344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6338.434329] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6343.149983] LustreError: 202761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.162393] LustreError: 202761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6347.292997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.726356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6352.123655] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6361.414564] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6367.707374] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6376.153604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.787330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6378.876908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6384.858721] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6393.839375] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6399.614612] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6409.255713] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6419.824957] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6424.583860] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6435.297106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6435.306432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.318708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6439.395563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6439.407359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6439.409795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.564156] Lustre: server umount lustre-OST0000 complete [ 6449.635407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6449.650485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6449.653291] Lustre: Skipped 1 previous similar message [ 6451.613539] Lustre: server umount lustre-MDT0000 complete [ 6454.772566] LustreError: 202762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.819260] LustreError: 202762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6456.412538] LustreError: 202743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399754 with bad export cookie 1371932738984209048 [ 6456.440042] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.782299] Lustre: server umount lustre-MDT0001 complete [ 6470.478308] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6473.100313] Key type lgssc unregistered [ 6473.411797] LNet: 207686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6473.431368] LNetError: 207686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6473.454745] LNet: Removed LNI 192.168.201.113@tcp [ 6474.183385] Key type .llcrypt unregistered [ 6474.185428] Key type ._llcrypt unregistered [ 6500.338731] Key type ._llcrypt registered [ 6500.340825] Key type .llcrypt registered [ 6500.425434] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6501.362048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.427687] alg: No test for adler32 (adler32-zlib) [ 6502.492444] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6502.828046] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6504.504278] Key type lgssc registered [ 6505.704173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6512.663458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6512.683390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.078363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6517.899811] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6524.397823] LustreError: 209102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.417073] LustreError: 209102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6526.050781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6526.433862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6530.767917] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6538.755184] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6544.219515] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6549.910396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.238336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6555.947358] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6559.501556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6563.489525] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6582.242925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6582.257262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6582.264316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.317304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.327044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.345751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6587.739934] Lustre: server umount lustre-OST0000 complete [ 6592.995866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6593.022962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6593.065641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6593.072469] Lustre: Skipped 1 previous similar message [ 6595.554122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6595.555255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.503939] Lustre: server umount lustre-MDT0000 complete [ 6600.675182] LustreError: 209103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.699828] LustreError: 209103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6602.929658] LustreError: 209083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399900 with bad export cookie 3549564234493970683 [ 6602.938357] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.945357] LustreError: 209083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6603.359462] Lustre: server umount lustre-MDT0001 complete [ 6615.293729] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6617.810456] Key type lgssc unregistered [ 6618.185601] LNet: 212785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6618.195207] LNetError: 212785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6618.206761] LNet: Removed LNI 192.168.201.113@tcp [ 6619.077192] Key type .llcrypt unregistered [ 6619.078903] Key type ._llcrypt unregistered [ 6639.604433] Key type ._llcrypt registered [ 6639.606622] Key type .llcrypt registered [ 6639.741531] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6640.995510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6641.026245] alg: No test for adler32 (adler32-zlib) [ 6642.066920] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6642.236536] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6643.904198] Key type lgssc registered [ 6645.007277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6654.442555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6654.487669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.116622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.356104] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6666.739721] LustreError: 214049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6666.768442] LustreError: 214049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6667.747628] LustreError: 214050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6669.914777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.215487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6675.139597] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6682.613952] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6688.096972] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6695.471348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6695.890319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6702.019654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6702.832068] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6710.178738] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6725.610508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.614575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.623103] Lustre: Skipped 1 previous similar message [ 6725.632170] Lustre: Skipped 1 previous similar message [ 6729.934590] Lustre: server umount lustre-OST0000 complete [ 6735.852032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.862635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6735.865961] Lustre: Skipped 1 previous similar message [ 6735.885472] Lustre: Skipped 1 previous similar message [ 6740.482964] Lustre: server umount lustre-MDT0000 complete [ 6740.961336] LustreError: 214045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.976501] LustreError: 214045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6744.523761] LustreError: 216547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400042 with bad export cookie 14737061410959768209 [ 6744.550150] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.847977] Lustre: server umount lustre-MDT0001 complete [ 6756.927774] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6759.611410] Key type lgssc unregistered [ 6759.883609] LNet: 217735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6759.905353] LNetError: 217735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6759.923220] LNet: Removed LNI 192.168.201.113@tcp [ 6760.596370] Key type .llcrypt unregistered [ 6760.600672] Key type ._llcrypt unregistered [ 6780.488272] Key type ._llcrypt registered [ 6780.494974] Key type .llcrypt registered [ 6780.646103] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6781.874490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6782.073207] alg: No test for adler32 (adler32-zlib) [ 6783.130661] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 6783.364102] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6785.048207] Key type lgssc registered [ 6786.326950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6795.271749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6795.301326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.910953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.031728] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6806.504615] LustreError: 219020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.519201] LustreError: 219020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6811.622453] LustreError: 219019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.017743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.326969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6817.993556] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6825.352451] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6831.169512] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6839.235383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6839.782130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.867352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6847.814621] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6855.778813] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6865.372671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6865.455319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6865.685096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6872.102291] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6880.577912] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6882.688472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6894.557321] Lustre: Permanently deactivating lustre-OST0001 [ 6894.585325] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6901.875588] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6910.927324] Lustre: Permanently reactivating lustre-OST0001 [ 6910.960474] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6910.968285] Lustre: Skipped 2 previous similar messages [ 6917.677650] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.694150] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6917.701402] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6917.719232] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6917.771363] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6926.484555] Lustre: server umount lustre-OST0001 complete [ 6928.352752] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6928.360666] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6928.366158] LustreError: 220841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6928.372520] Lustre: Skipped 2 previous similar messages [ 6933.474923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.485753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.489332] LustreError: 220840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6933.504359] LustreError: 220840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6935.919821] Lustre: server umount lustre-OST0000 complete [ 6943.714853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.724441] Lustre: Skipped 1 previous similar message [ 6943.732597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.736224] Lustre: Skipped 2 previous similar messages [ 6946.623503] Lustre: server umount lustre-MDT0000 complete [ 6948.834169] LustreError: 219016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.848548] LustreError: 219016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6950.346425] LustreError: 219000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400248 with bad export cookie 5912740491355416339 [ 6950.347414] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6950.359391] LustreError: 219000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6950.802237] Lustre: server umount lustre-MDT0001 complete [ 6961.717372] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6964.260404] Key type lgssc unregistered [ 6964.528943] LNet: 224109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6964.537806] LNetError: 224109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6964.580893] LNet: Removed LNI 192.168.201.113@tcp [ 6965.358127] Key type .llcrypt unregistered [ 6965.359681] Key type ._llcrypt unregistered [ 7007.679266] Key type ._llcrypt registered [ 7007.683208] Key type .llcrypt registered [ 7007.821808] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7008.887905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7009.019151] alg: No test for adler32 (adler32-zlib) [ 7010.059745] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7010.351443] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7012.049585] Key type lgssc registered [ 7013.395427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7022.293517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7022.318512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.659361] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7023.673257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7023.867216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.646516] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7040.421499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.608604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7040.684835] Lustre: 226313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7040.973215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7046.780390] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7056.052942] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7063.347469] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7070.697712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7070.826688] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7071.038639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.855505] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7079.580876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7085.748121] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7089.593399] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7096.270708] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7103.423830] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7112.406681] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7122.547273] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7143.739922] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7143.757193] Lustre: Skipped 2 previous similar messages [ 7176.119558] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7176.131062] Lustre: Skipped 3 previous similar messages [ 7242.392905] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7242.421538] Lustre: Skipped 7 previous similar messages [ 7273.957229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.972127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.987517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.475098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.484990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.504937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.511640] Lustre: Skipped 1 previous similar message [ 7278.304698] Lustre: server umount lustre-OST0000 complete [ 7284.706587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.712773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7284.720626] Lustre: Skipped 1 previous similar message [ 7288.926717] Lustre: server umount lustre-MDT0000 complete [ 7292.855383] LustreError: 226305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400590 with bad export cookie 8253866132195288424 [ 7292.856041] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.869450] LustreError: 226305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7293.181580] Lustre: server umount lustre-MDT0001 complete [ 7305.318240] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7308.180635] Key type lgssc unregistered [ 7308.445670] LNet: 231121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7308.456559] LNetError: 231121:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7308.467875] LNet: Removed LNI 192.168.201.113@tcp [ 7309.215747] Key type .llcrypt unregistered [ 7309.220837] Key type ._llcrypt unregistered [ 7332.595865] Key type ._llcrypt registered [ 7332.597908] Key type .llcrypt registered [ 7332.683424] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7333.730912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7333.795147] alg: No test for adler32 (adler32-zlib) [ 7334.950165] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7335.270076] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7336.904184] Key type lgssc registered [ 7337.966415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7346.329208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7346.351134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.947283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7352.458894] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7357.411895] LustreError: 232538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7357.437677] LustreError: 232538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7361.367971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7361.666246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7366.464849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7375.012851] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7381.840656] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7389.698493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.172930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7392.236645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7396.717586] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7404.281277] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7421.924512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7421.929635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.942376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7425.851793] Lustre: server umount lustre-OST0000 complete [ 7432.163092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.170196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.178072] Lustre: Skipped 2 previous similar messages [ 7432.196531] Lustre: Skipped 2 previous similar messages [ 7435.872985] Lustre: server umount lustre-MDT0000 complete [ 7437.287793] LustreError: 232538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.320685] LustreError: 232538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7439.342772] LustreError: 232517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400737 with bad export cookie 5440024275851220793 [ 7439.345111] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.360099] LustreError: 232517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7439.755196] Lustre: server umount lustre-MDT0001 complete [ 7450.960890] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7453.976238] Key type lgssc unregistered [ 7454.260753] LNet: 236227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7454.269826] LNetError: 236227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7455.340792] LNet: Removed LNI 192.168.201.113@tcp [ 7456.287075] Key type .llcrypt unregistered [ 7456.290439] Key type ._llcrypt unregistered [ 7482.569867] Key type ._llcrypt registered [ 7482.574959] Key type .llcrypt registered [ 7482.717350] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7484.152906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7484.178160] alg: No test for adler32 (adler32-zlib) [ 7485.181801] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7485.383810] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7487.024357] Key type lgssc registered [ 7488.532399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7761.053502] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7764.239447] Key type lgssc unregistered [ 7764.585553] LNet: 238251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7764.591343] LNetError: 238251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7764.608703] LNet: Removed LNI 192.168.201.113@tcp [ 7765.548651] Key type .llcrypt unregistered [ 7765.556353] Key type ._llcrypt unregistered [ 7798.754569] Key type ._llcrypt registered [ 7798.758617] Key type .llcrypt registered [ 7798.843163] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 7819.274389] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7820.804431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7820.871961] alg: No test for adler32 (adler32-zlib) [ 7821.999682] Lustre: Lustre: Build Version: 2.17.54_107_g42be189 [ 7822.280101] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7823.984196] Key type lgssc registered [ 7825.336510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7862.964293] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7574 sec ========== 08:19:18 (1787401158) [ 7864.818646] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:19:20 (1787401160) === [ 7869.604523] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:19:24 (1787401164) === [ 7924.726258] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7928.385616] Key type lgssc unregistered [ 7928.573960] LNet: 243500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7928.584237] LNetError: 243500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7928.598729] LNet: Removed LNI 192.168.201.113@tcp [ 7929.723314] Key type .llcrypt unregistered [ 7929.728775] Key type ._llcrypt unregistered