[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481933220 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003373] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011008] pid_max: default: 32768 minimum: 301 [ 0.012132] LSM: Security Framework initializing [ 0.013059] Yama: becoming mindful. [ 0.014044] SELinux: Initializing. [ 0.015063] *** VALIDATE selinux *** [ 0.023190] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027760] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031038] *** VALIDATE tmpfs *** [ 0.032404] *** VALIDATE proc *** [ 0.033230] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036209] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039025] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043183] debug: unmapping init [mem 0xffffffffad259000-0xffffffffad260fff] [ 0.045152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046604] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047021] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054267] rcu: Hierarchical SRCU implementation. [ 0.056417] smp: Bringing up secondary CPUs ... [ 0.057520] x86: Booting SMP configuration: [ 0.058021] .... node #0, CPUs: #1 #2 #3 [ 0.061335] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.181072] node 0 deferred pages initialised in 115ms [ 0.185009] devtmpfs: initialized [ 0.186229] x86/mm: Memory block size: 128MB [ 0.188601] gcov: version magic: 0x41383552 [ 0.189578] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.194119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.197438] pinctrl core: initialized pinctrl subsystem [ 0.200248] [ 0.200909] ************************************************************* [ 0.203021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.206022] ** ** [ 0.209022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.211018] ** ** [ 0.214021] ** This means that this kernel is built to expose internal ** [ 0.218023] ** IOMMU data structures, which may compromise security on ** [ 0.220018] ** your system. ** [ 0.222019] ** ** [ 0.224023] ** If you see this message and you are not debugging the ** [ 0.226020] ** kernel, report this immediately to your vendor! ** [ 0.228023] ** ** [ 0.232023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.234016] ************************************************************* [ 0.236799] NET: Registered protocol family 16 [ 0.238531] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.241084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.244084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.248133] cpuidle: using governor menu [ 0.249691] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.252777] PCI: Using configuration type 1 for base access [ 0.255123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.264119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270073] cryptd: max_cpu_qlen set to 1000 [ 0.272262] ACPI: Added _OSI(Module Device) [ 0.274033] ACPI: Added _OSI(Processor Device) [ 0.276016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.278013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.281000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.289258] ACPI: Interpreter enabled [ 0.290052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.292013] ACPI: Using IOAPIC for interrupt routing [ 0.293110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.296338] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.307033] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.309045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.312025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.316095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.321424] acpiphp: Slot [2] registered [ 0.322121] acpiphp: Slot [5] registered [ 0.324128] acpiphp: Slot [6] registered [ 0.325113] acpiphp: Slot [7] registered [ 0.326096] acpiphp: Slot [8] registered [ 0.327104] acpiphp: Slot [9] registered [ 0.328099] acpiphp: Slot [10] registered [ 0.330095] acpiphp: Slot [3] registered [ 0.331116] acpiphp: Slot [4] registered [ 0.333127] acpiphp: Slot [11] registered [ 0.335108] acpiphp: Slot [12] registered [ 0.336122] acpiphp: Slot [13] registered [ 0.338143] acpiphp: Slot [14] registered [ 0.339103] acpiphp: Slot [15] registered [ 0.341113] acpiphp: Slot [16] registered [ 0.342077] acpiphp: Slot [17] registered [ 0.343094] acpiphp: Slot [18] registered [ 0.345138] acpiphp: Slot [19] registered [ 0.346120] acpiphp: Slot [20] registered [ 0.348128] acpiphp: Slot [21] registered [ 0.350135] acpiphp: Slot [22] registered [ 0.351137] acpiphp: Slot [23] registered [ 0.353133] acpiphp: Slot [24] registered [ 0.354149] acpiphp: Slot [25] registered [ 0.356086] acpiphp: Slot [26] registered [ 0.357160] acpiphp: Slot [27] registered [ 0.359131] acpiphp: Slot [28] registered [ 0.361124] acpiphp: Slot [29] registered [ 0.362150] acpiphp: Slot [30] registered [ 0.364123] acpiphp: Slot [31] registered [ 0.365082] PCI host bridge to bus 0000:00 [ 0.367029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.371030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.376025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.379033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.382167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.383884] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.387246] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397713] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.402571] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.406021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.409026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.411019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.415699] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.418911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.426010] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.431020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.445021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.450845] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456718] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.471020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.491021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.502474] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.517062] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.535019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.545446] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.558020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.566021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.594023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.607383] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.618014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.626015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.641027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.653121] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.660025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.701620] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.711060] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.719019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.737024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.749513] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.751390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.753439] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.757425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.762343] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.768713] iommu: Default domain type: Passthrough [ 0.770536] SCSI subsystem initialized [ 0.772169] ACPI: bus type USB registered [ 0.774148] usbcore: registered new interface driver usbfs [ 0.776091] usbcore: registered new interface driver hub [ 0.777108] usbcore: registered new device driver usb [ 0.780209] pps_core: LinuxPPS API ver. 1 registered [ 0.782016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.785117] PTP clock support registered [ 0.788164] EDAC MC: Ver: 3.0.0 [ 0.790454] PCI: Using ACPI for IRQ routing [ 0.792973] NetLabel: Initializing [ 0.794017] NetLabel: domain hash size = 128 [ 0.796015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.799114] NetLabel: unlabeled traffic allowed by default [ 0.802061] vgaarb: loaded [ 0.803332] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.805014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.812000] clocksource: Switched to clocksource kvm-clock [ 0.920375] VFS: Disk quotas dquot_6.6.0 [ 0.922254] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.925501] *** VALIDATE ramfs *** [ 0.927097] *** VALIDATE hugetlbfs *** [ 0.929718] pnp: PnP ACPI init [ 0.932355] pnp: PnP ACPI: found 6 devices [ 0.949665] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.953424] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.955991] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.959488] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.962821] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.966105] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.969658] NET: Registered protocol family 2 [ 0.972870] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.978440] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.982359] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.986790] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.989993] TCP: Hash tables configured (established 65536 bind 65536) [ 0.992223] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.994799] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.996858] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.998976] NET: Registered protocol family 1 [ 1.001085] RPC: Registered named UNIX socket transport module. [ 1.003754] RPC: Registered udp transport module. [ 1.006388] RPC: Registered tcp transport module. [ 1.007881] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.010870] NET: Registered protocol family 44 [ 1.012868] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.015402] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.017858] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.020648] PCI: CLS 0 bytes, default 64 [ 1.022527] Unpacking initramfs... [ 2.441360] debug: unmapping init [mem 0xffff9037bcc54000-0xffff9037bffbffff] [ 2.447190] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.449969] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.453488] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.944070] Initialise system trusted keyrings [ 2.946287] Key type blacklist registered [ 2.948656] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.957881] zbud: loaded [ 2.961142] *** VALIDATE nfs *** [ 2.962550] *** VALIDATE nfs4 *** [ 2.964144] pstore: using deflate compression [ 2.968518] Platform Keyring initialized [ 3.074853] NET: Registered protocol family 38 [ 3.077239] Key type asymmetric registered [ 3.079042] Asymmetric key parser 'x509' registered [ 3.080922] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.084248] io scheduler mq-deadline registered [ 3.086179] io scheduler kyber registered [ 3.088059] io scheduler bfq registered [ 3.090197] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.093959] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.097229] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.100438] ACPI: Power Button [PWRF] [ 3.105949] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.112736] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.124865] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.133523] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.153936] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.183728] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.211507] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.217259] Non-volatile memory driver v1.3 [ 3.219050] Linux agpgart interface v0.103 [ 3.250699] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.253099] vda: detected capacity change from 0 to 71311360 [ 3.267509] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.270016] vdb: detected capacity change from 0 to 1073741824 [ 3.283766] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.286119] vdc: detected capacity change from 0 to 2621440000 [ 3.301105] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.304525] vdd: detected capacity change from 0 to 2621440000 [ 3.322756] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.325879] vde: detected capacity change from 0 to 4294967296 [ 3.343438] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.346910] vdf: detected capacity change from 0 to 4294967296 [ 3.353924] libphy: Fixed MDIO Bus: probed [ 3.366279] usbcore: registered new interface driver usbserial_generic [ 3.369100] usbserial: USB Serial support registered for generic [ 3.371651] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.376475] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.378799] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.382182] mousedev: PS/2 mouse device common for all mice [ 3.385247] rtc_cmos 00:05: RTC can wake from S4 [ 3.387986] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.388227] rtc_cmos 00:05: registered as rtc0 [ 3.392840] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.393666] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.401148] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.401433] intel_pstate: CPU model not supported [ 3.407075] hid: raw HID events driver (C) Jiri Kosina [ 3.409348] usbcore: registered new interface driver usbhid [ 3.411601] usbhid: USB HID core driver [ 3.413211] drop_monitor: Initializing network drop monitor service [ 3.415774] Initializing XFRM netlink socket [ 3.417893] NET: Registered protocol family 10 [ 3.421065] Segment Routing with IPv6 [ 3.422504] NET: Registered protocol family 17 [ 3.425382] mpls_gso: MPLS GSO support [ 3.431627] RAS: Correctable Errors collector initialized. [ 3.434154] AVX version of gcm_enc/dec engaged. [ 3.436072] AES CTR mode by8 optimization enabled [ 3.515080] sched_clock: Marking stable (3515042879, 0)->(4441794116, -926751237) [ 3.519248] registered taskstats version 1 [ 3.522161] Loading compiled-in X.509 certificates [ 3.524489] zswap: loaded using pool lzo/zbud [ 3.552986] Key type big_key registered [ 3.568229] Key type encrypted registered [ 3.570304] ima: No TPM chip found, activating TPM-bypass! [ 3.572394] ima: Allocated hash algorithm: sha1 [ 3.574209] ima: No architecture policies found [ 3.576102] evm: Initialising EVM extended attributes: [ 3.578020] evm: security.selinux [ 3.579233] evm: security.ima [ 3.580576] evm: security.capability [ 3.581814] evm: HMAC attrs: 0x1 [ 3.584689] rtc_cmos 00:05: setting system clock to 2026-06-16 03:58:55 UTC (1781582335) [ 3.591274] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.594341] debug: unmapping init [mem 0xffffffffacf82000-0xffffffffad258fff] [ 3.603385] Write protecting the kernel read-only data: 28672k [ 3.607207] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 3.610241] debug: unmapping init [mem 0xffffffffabf14000-0xffffffffabffffff] [ 3.649415] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.660063] systemd[1]: Detected virtualization kvm. [ 3.662462] systemd[1]: Detected architecture x86-64. [ 3.664934] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.691906] systemd[1]: No hostname configured. [ 3.694088] systemd[1]: Set hostname to . [ 3.696490] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699325] systemd[1]: Initializing machine ID from random generator. [ 3.745625] random: ln: uninitialized urandom read (6 bytes read) [ 3.850893] random: systemd: uninitialized urandom read (16 bytes read) [ 3.854223] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.858947] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.863745] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [[ 4.094605] random: fast init done  OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.519584] device-mapper: uevent: version 1.0.3 [ 4.521879] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.372861] virtio_net virtio0 ens2: renamed from eth0 [ 5.382241] scsi host0: ata_piix [ 5.428636] scsi host1: ata_piix [ 5.430940] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.433364] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.193547] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.088161] random: crng init done [ 10.089768] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.628768] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.785218] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.029158] SELinux: Disabled at runtime. [ 12.091167] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.101517] systemd[1]: Detected virtualization kvm. [ 12.103655] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.580452] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.584311] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.588178] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.593076] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.597146] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.603972] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.607547] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ 12.738805] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.973127] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.324241] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.385227] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.470191] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.481347] EDAC sbridge: Ver: 1.1.2 [ 15.113314] Key type dns_resolver registered [ 15.416070] NFS: Registering the id_resolver key type [ 15.419627] Key type id_resolver registered [ 15.421410] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg352-server login: [ 74.171414] libcfs: loading out-of-tree module taints kernel. [ 74.242927] Key type ._llcrypt registered [ 74.244699] Key type .llcrypt registered [ 74.348885] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 91.028275] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 92.438233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.457821] alg: No test for adler32 (adler32-zlib) [ 93.757845] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 94.425808] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 96.127681] Key type lgssc registered [ 97.884204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.984394] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.332355] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 171.313041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.357458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.639798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.668632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.740208] Lustre: lustre-MDT0000: new disk, initializing [ 172.802842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.812875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 176.424499] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.954461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.047588] Lustre: 6531:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.079263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.083769] Lustre: Skipped 1 previous similar message [ 189.226193] Lustre: lustre-MDT0001: new disk, initializing [ 189.315968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.339958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 189.354595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 193.675295] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.008298] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 208.057238] hrtimer: interrupt took 4430802 ns [ 209.260853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.552604] Lustre: lustre-OST0000: new disk, initializing [ 209.558610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 209.578315] Lustre: 8468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.659197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.341949] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.113711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.125420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.205486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.797192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.056458] Lustre: lustre-OST0001: new disk, initializing [ 232.061535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 232.073312] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.214627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.437496] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.765937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.793675] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.845979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.063783] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.403431] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.184691] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing check_logdir /tmp/testlogs/ [ 269.667673] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing yml_node [ 274.032686] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 276.636535] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 279.421910] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 281.164290] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 00:03:30 EDT 2026 [ 298.174919] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 299.916616] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 317.408547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 317.419449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.427158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.944888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.944912] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.440519] Lustre: server umount lustre-MDT0000 complete [ 324.575594] LustreError: 7477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781582656 with bad export cookie 6429413677755533035 [ 324.580083] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.585600] LustreError: 7477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 329.201623] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.202235] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.224159] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 329.224877] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.224881] Lustre: Skipped 3 previous similar messages [ 329.276974] Lustre: Skipped 3 previous similar messages [ 331.025458] Lustre: server umount lustre-MDT0001 complete [ 335.020985] Lustre: server umount lustre-OST0000 complete [ 339.371106] Lustre: server umount lustre-OST0001 complete [ 346.237106] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 356.739071] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 395.204883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.543971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.602108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.717827] Lustre: lustre-MDT0000: new disk, initializing [ 395.829966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.874728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.663295] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 411.804073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.988674] Lustre: 15095:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.058681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.062504] Lustre: Skipped 1 previous similar message [ 412.157271] Lustre: lustre-MDT0001: new disk, initializing [ 412.281916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.346185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.369107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.237235] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 427.670688] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.124932] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 441.726770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 442.079432] Lustre: lustre-OST0000: new disk, initializing [ 442.087565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 442.105198] Lustre: 17063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 442.244741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.015552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.040738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.175547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.400460] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 459.345355] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.475780] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.672089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.167955] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.379273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.104918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.117309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.150696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.160516] Lustre: Skipped 1 previous similar message [ 477.875953] Lustre: server umount lustre-OST0000 complete [ 483.810072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.818130] LustreError: Skipped 1 previous similar message [ 483.823315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.850636] Lustre: Skipped 1 previous similar message [ 483.856768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 483.859810] Lustre: Skipped 1 previous similar message [ 488.214894] Lustre: server umount lustre-MDT0000 complete [ 490.474783] LustreError: 15101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.500351] LustreError: 15101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 492.215598] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781582824 with bad export cookie 6429413677755534568 [ 492.221148] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.227316] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 492.603872] Lustre: server umount lustre-MDT0001 complete [ 496.161238] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 00:07:06 (1781582826) [ 503.230802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.767576] LustreError: 19325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.865297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.366714] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 508.898211] LustreError: 19326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.026105] LustreError: 19325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.111363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.519732] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 532.698665] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 539.553951] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 547.032667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 553.696727] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 563.006367] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 573.923114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.933772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.948017] Lustre: Skipped 1 previous similar message [ 573.952295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.954844] Lustre: Skipped 2 previous similar messages [ 579.780886] Lustre: server umount lustre-OST0000 complete [ 589.957170] Lustre: server umount lustre-MDT0000 complete [ 590.306611] LustreError: 19322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.324229] LustreError: 19322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 593.968826] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781582925 with bad export cookie 6429413677755535450 [ 593.978199] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.981714] LustreError: 19307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 594.312782] Lustre: server umount lustre-MDT0001 complete [ 606.322071] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 609.060254] Key type lgssc unregistered [ 609.334124] LNet: 22860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.339635] LNetError: 22860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.352792] LNet: Removed LNI 192.168.203.152@tcp [ 610.303286] Key type .llcrypt unregistered [ 610.305382] Key type ._llcrypt unregistered [ 634.377695] Key type ._llcrypt registered [ 634.380617] Key type .llcrypt registered [ 634.482720] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 635.951627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.984253] alg: No test for adler32 (adler32-zlib) [ 637.068225] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 637.309109] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 639.079610] Key type lgssc registered [ 640.292545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 647.744571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 647.761700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.242407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.536662] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 659.431150] LustreError: 24286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.443619] LustreError: 24286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 661.867076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.197141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 666.260084] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 673.805814] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 679.518672] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 687.249650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 687.621421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.262522] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 694.847924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 701.590958] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 715.235463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.244207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.271758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 717.403911] Lustre: server umount lustre-OST0000 complete [ 723.428989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.435554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 723.436199] Lustre: Skipped 1 previous similar message [ 723.446472] Lustre: Skipped 1 previous similar message [ 728.544762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.560026] Lustre: Skipped 1 previous similar message [ 733.668101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.676528] Lustre: Skipped 1 previous similar message [ 734.689085] Lustre: 23597:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781583050/real 1781583050] req@000000001da24dad x1868125143332480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781583066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 734.733743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.199822] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 735.508804] Lustre: server umount lustre-MDT0000 complete [ 735.778855] Lustre: 23597:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781583051/real 1781583051] req@000000004d0b50af x1868125143332992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781583067 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 739.048056] LustreError: 24267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583070 with bad export cookie 8447385396457924503 [ 739.057333] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.058593] LustreError: 24267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 739.518244] Lustre: server umount lustre-MDT0001 complete [ 751.307263] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 753.760590] Key type lgssc unregistered [ 754.169738] LNet: 28260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.187944] LNetError: 28260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.219839] LNet: Removed LNI 192.168.203.152@tcp [ 754.994197] Key type .llcrypt unregistered [ 754.998658] Key type ._llcrypt unregistered [ 781.920255] Key type ._llcrypt registered [ 781.922320] Key type .llcrypt registered [ 782.100123] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 783.450727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.474729] alg: No test for adler32 (adler32-zlib) [ 784.624877] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 784.915450] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 786.623163] Key type lgssc registered [ 787.597419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 795.156429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 795.223953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.980965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 801.653685] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 806.369345] LustreError: 29675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.381366] LustreError: 29675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 811.363243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.494141] LustreError: 29674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.853565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.091277] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 823.258455] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 828.969792] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 845.568651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.159446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.133691] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 861.172097] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.619241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 871.916056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.918742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.923602] LustreError: Skipped 1 previous similar message [ 871.926498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.944653] Lustre: Skipped 1 previous similar message [ 871.957960] Lustre: Skipped 1 previous similar message [ 873.675451] Lustre: server umount lustre-OST0000 complete [ 878.560480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.574365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.592287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 882.148093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 882.148444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.479361] Lustre: server umount lustre-MDT0000 complete [ 887.032866] LustreError: 30408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583218 with bad export cookie 3926236018761073155 [ 887.049138] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 887.050243] LustreError: 30408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 887.399840] Lustre: server umount lustre-MDT0001 complete [ 898.910645] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 901.596362] Key type lgssc unregistered [ 901.832024] LNet: 33552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.838251] LNetError: 33552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.849314] LNet: Removed LNI 192.168.203.152@tcp [ 902.863450] Key type .llcrypt unregistered [ 902.865341] Key type ._llcrypt unregistered [ 929.078961] Key type ._llcrypt registered [ 929.080939] Key type .llcrypt registered [ 929.242203] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 930.137514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 930.259245] alg: No test for adler32 (adler32-zlib) [ 931.310217] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 931.609095] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 933.375209] Key type lgssc registered [ 934.801190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.187644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.223173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.023397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.213385] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 954.343394] LustreError: 34968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.367288] LustreError: 34968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 959.470447] LustreError: 34969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.238130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.493824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 965.125500] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 974.893949] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 981.309334] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 987.614149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.999477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.068994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 993.541539] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1001.819750] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1015.782219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1015.795053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.815109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.806541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.811324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.766208] Lustre: server umount lustre-OST0000 complete [ 1027.041338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.046161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.057508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.072323] Lustre: Skipped 1 previous similar message [ 1027.088167] Lustre: Skipped 2 previous similar messages [ 1032.175723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.181614] Lustre: Skipped 1 previous similar message [ 1036.257570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.269511] Lustre: Skipped 1 previous similar message [ 1040.351221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.518511] Lustre: server umount lustre-MDT0000 complete [ 1041.376831] LustreError: 34969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.407712] LustreError: 34969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1044.757483] LustreError: 34951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583376 with bad export cookie 4651580561285957835 [ 1044.768965] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.770525] LustreError: 34951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1045.131679] Lustre: server umount lustre-MDT0001 complete [ 1056.504587] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1059.321667] Key type lgssc unregistered [ 1059.682690] LNet: 38526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.696190] LNetError: 38526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.709066] LNet: Removed LNI 192.168.203.152@tcp [ 1060.503549] Key type .llcrypt unregistered [ 1060.511302] Key type ._llcrypt unregistered [ 1082.667197] Key type ._llcrypt registered [ 1082.670043] Key type .llcrypt registered [ 1082.773742] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1083.748227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.805404] alg: No test for adler32 (adler32-zlib) [ 1084.828435] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1085.043370] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1086.687180] Key type lgssc registered [ 1087.688582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.077523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.090713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.550587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.198930] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1106.918878] LustreError: 39944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.943695] LustreError: 39944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1109.357426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.757348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1114.397536] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1121.701908] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1128.353733] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1136.004582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1136.338924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.524226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1143.668556] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1152.834332] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1162.213036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.228919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.260513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.292165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.292711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.310489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.382699] Lustre: lustre-OST0000: Not available for connect from 192.168.203.52@tcp (stopping) [ 1166.390674] Lustre: Skipped 1 previous similar message [ 1167.965696] Lustre: server umount lustre-OST0000 complete [ 1187.279141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.288896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.622276] Lustre: server umount lustre-MDT0000 complete [ 1195.488746] LustreError: 39943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.682513] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583528 with bad export cookie 8774953183701049658 [ 1196.690843] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.154657] Lustre: server umount lustre-MDT0001 complete [ 1212.703777] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1215.272924] Key type lgssc unregistered [ 1215.554872] LNet: 43651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.569345] LNetError: 43651:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.583921] LNet: Removed LNI 192.168.203.152@tcp [ 1216.529632] Key type .llcrypt unregistered [ 1216.532599] Key type ._llcrypt unregistered [ 1240.147815] Key type ._llcrypt registered [ 1240.150443] Key type .llcrypt registered [ 1240.260917] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1241.489143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1241.591239] alg: No test for adler32 (adler32-zlib) [ 1242.720915] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1243.024791] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1244.759563] Key type lgssc registered [ 1246.060346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.817118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.841142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.450700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.545542] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1267.684716] LustreError: 45070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.717160] LustreError: 45070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1272.069541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.561632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.816423] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1283.419598] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.479824] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1296.527253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.864811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.044673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1303.206616] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1310.609715] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1317.758203] Lustre: server umount lustre-MDT0000 complete [ 1318.376585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.384571] LustreError: 45799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.389363] Lustre: Skipped 1 previous similar message [ 1318.414962] LustreError: 45799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1322.357432] LustreError: 45051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583654 with bad export cookie 17624593238625912712 [ 1322.395249] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.396498] LustreError: 45051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1323.490502] LustreError: 46966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.491269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.531414] LustreError: 46966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1323.579103] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 1323.582921] Lustre: Skipped 1 previous similar message [ 1323.597433] Lustre: Skipped 1 previous similar message [ 1327.523917] LustreError: 45065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.524739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1327.559886] LustreError: 45065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1328.691954] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.52@tcp (stopping) [ 1328.978187] Lustre: server umount lustre-MDT0001 complete [ 1350.550867] Lustre: server umount lustre-OST0000 complete [ 1367.353668] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1369.808674] Key type lgssc unregistered [ 1370.073345] LNet: 48929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.084315] LNetError: 48929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.102268] LNet: Removed LNI 192.168.203.152@tcp [ 1370.864448] Key type .llcrypt unregistered [ 1370.866554] Key type ._llcrypt unregistered [ 1394.848388] Key type ._llcrypt registered [ 1394.858931] Key type .llcrypt registered [ 1394.969902] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1396.033925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.046200] alg: No test for adler32 (adler32-zlib) [ 1397.045174] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1397.260194] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1399.057196] Key type lgssc registered [ 1400.122845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1406.726546] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1406.740882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1422.495349] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1427.615204] LustreError: 50316:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 1454.304515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1459.513115] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1466.277182] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1633.689420] Lustre: server umount lustre-OST0000 complete [ 1649.311215] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1651.916978] Key type lgssc unregistered [ 1652.256412] LNet: 51991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.263819] LNetError: 51991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.275122] LNet: Removed LNI 192.168.203.152@tcp [ 1652.989536] Key type .llcrypt unregistered [ 1652.994078] Key type ._llcrypt unregistered [ 1674.944734] Key type ._llcrypt registered [ 1674.947826] Key type .llcrypt registered [ 1675.074373] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1676.125057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.340362] alg: No test for adler32 (adler32-zlib) [ 1677.428398] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1677.647785] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1679.303234] Key type lgssc registered [ 1680.507873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1687.247808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1687.273092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.872717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.852831] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1698.278084] LustreError: 53406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.298139] LustreError: 53406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1700.475995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1700.873982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1704.757136] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1712.415839] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1718.784968] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1725.433542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.772043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.625634] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1734.031965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1742.014607] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1745.512132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.518232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.529881] Lustre: Skipped 1 previous similar message [ 1750.713588] Lustre: server umount lustre-OST0000 complete [ 1756.133198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.147848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1761.192803] Lustre: server umount lustre-MDT0000 complete [ 1761.256141] LustreError: 53401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.293641] LustreError: 53401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1765.879053] LustreError: 53385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584097 with bad export cookie 7910822532448680465 [ 1765.886883] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1765.894164] LustreError: 53385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1766.265222] Lustre: server umount lustre-MDT0001 complete [ 1779.429203] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1782.205985] Key type lgssc unregistered [ 1782.473112] LNet: 56936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.482908] LNetError: 56936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.508138] LNet: Removed LNI 192.168.203.152@tcp [ 1783.386211] Key type .llcrypt unregistered [ 1783.387967] Key type ._llcrypt unregistered [ 1805.597097] Key type ._llcrypt registered [ 1805.599636] Key type .llcrypt registered [ 1805.711132] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 1806.721802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.923404] alg: No test for adler32 (adler32-zlib) [ 1807.940937] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1808.149592] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 1809.823893] Key type lgssc registered [ 1810.891191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.293476] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1818.315164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1834.016539] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1839.139201] LustreError: 58325:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 1865.823241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.305258] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1881.069377] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1888.772425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.182631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.267324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1894.372578] LustreError: 58350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.419175] LustreError: 58350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1894.847422] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1899.488267] LustreError: 58351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.516831] LustreError: 58350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.537816] LustreError: 58350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1903.513308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.989318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1908.989687] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 1917.173984] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1923.988934] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1929.709893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.726303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.726763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.739873] Lustre: Skipped 1 previous similar message [ 1929.763338] Lustre: Skipped 1 previous similar message [ 1932.622648] Lustre: server umount lustre-OST0000 complete [ 1959.403203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.415587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.431332] Lustre: Skipped 1 previous similar message [ 1959.444426] Lustre: Skipped 1 previous similar message [ 1962.378950] Lustre: server umount lustre-MDT0000 complete [ 1964.530707] LustreError: 59361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.553018] LustreError: 59361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1965.958404] LustreError: 58331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584297 with bad export cookie 15607668540389089059 [ 1965.964424] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.972701] LustreError: 58331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1980.383140] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1980.893143] Lustre: server umount lustre-MDT0001 complete [ 1993.886841] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 1997.667373] Key type lgssc unregistered [ 1998.145444] LNet: 62076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.161626] LNetError: 62076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.190973] LNet: Removed LNI 192.168.203.152@tcp [ 1999.121957] Key type .llcrypt unregistered [ 1999.126628] Key type ._llcrypt unregistered [ 2026.071873] Key type ._llcrypt registered [ 2026.078249] Key type .llcrypt registered [ 2026.179046] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2027.360707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.379587] alg: No test for adler32 (adler32-zlib) [ 2028.516489] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2028.740723] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2030.415193] Key type lgssc registered [ 2031.637991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.183600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.198860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.973922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2047.368363] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2051.558894] LustreError: 63491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.593038] LustreError: 63491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2056.678699] LustreError: 63490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.297201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.908121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2062.871586] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2070.277292] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2076.837983] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2083.250953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2083.656988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.772494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2089.527815] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2097.439446] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2116.422387] Lustre: server umount lustre-OST0000 complete [ 2122.720446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.722604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.734840] Lustre: Skipped 1 previous similar message [ 2127.846557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2127.849490] Lustre: Skipped 1 previous similar message [ 2132.963810] LustreError: 63491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.989771] LustreError: 63491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2133.083963] Lustre: server umount lustre-MDT0000 complete [ 2133.984684] Lustre: 62802:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781584449/real 1781584449] req@000000001da24dad x1868126601903104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781584465 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2133.988885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.037952] Lustre: 62802:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2137.619706] LustreError: 63472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584469 with bad export cookie 1609602483046872446 [ 2137.622990] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2137.639857] LustreError: 63472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2137.930103] Lustre: server umount lustre-MDT0001 complete [ 2149.363256] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2152.308375] Key type lgssc unregistered [ 2152.664920] LNet: 67043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.676503] LNetError: 67043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2152.690694] LNet: Removed LNI 192.168.203.152@tcp [ 2153.482988] Key type .llcrypt unregistered [ 2153.485264] Key type ._llcrypt unregistered [ 2177.046537] Key type ._llcrypt registered [ 2177.048934] Key type .llcrypt registered [ 2177.161815] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2178.038105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.083075] alg: No test for adler32 (adler32-zlib) [ 2179.077095] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2179.273617] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2180.935460] Key type lgssc registered [ 2181.933901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2190.262139] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 00:35:20 (1781584520) [ 2196.310212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2196.335178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.863438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.369970] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2207.202280] LustreError: 68728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.232338] LustreError: 68728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2210.341346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.606077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2214.493463] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2221.324565] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2226.822769] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2233.186830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2233.504513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.269567] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2242.733155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2246.351358] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2251.817510] Lustre: Failing over lustre-MDT0000 [ 2253.281959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.301920] LustreError: 70071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.318780] Lustre: Skipped 2 previous similar messages [ 2253.348980] LustreError: 70071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2254.039855] Lustre: server umount lustre-MDT0000 complete [ 2258.401634] LustreError: 69454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.434065] LustreError: 69454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2262.677437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.751816] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.843727] Lustre: *** cfs_fail_loc=135, val=0*** [ 2262.986710] LustreError: 71814:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2262.997458] LustreError: 71814:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.152@tcp: cfg command failed: rc = -2 [ 2263.013811] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2263.013811] [ 2263.033655] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2263.052418] LustreError: 71802:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2263.068903] LustreError: 71802:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2263.077836] LustreError: 71802:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2263.080934] LustreError: 71802:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2263.116088] Lustre: server umount lustre-MDT0000 complete [ 2263.118171] LustreError: 71802:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2263.528236] LustreError: 68727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.551627] LustreError: 68727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2268.643138] LustreError: 68728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.668509] LustreError: 68728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2269.850496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.036743] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2270.055396] LustreError: 68719:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2270.239345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2270.286053] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2273.795292] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2274.705444] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2275.312561] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2275.333320] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2275.389556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2285.542516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.548802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.553954] Lustre: Skipped 1 previous similar message [ 2289.185315] Lustre: server umount lustre-OST0000 complete [ 2295.776979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.778257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2295.781811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.781819] Lustre: Skipped 1 previous similar message [ 2295.789374] Lustre: Skipped 1 previous similar message [ 2298.759127] Lustre: server umount lustre-MDT0000 complete [ 2300.898385] LustreError: 70070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.915432] LustreError: 70070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2302.042446] LustreError: 68707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584633 with bad export cookie 14105010257431667645 [ 2302.055578] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2302.374149] Lustre: server umount lustre-MDT0001 complete [ 2315.215580] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2318.015905] Key type lgssc unregistered [ 2318.305345] LNet: 73596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.316678] LNetError: 73596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.329131] LNet: Removed LNI 192.168.203.152@tcp [ 2319.177505] Key type .llcrypt unregistered [ 2319.182069] Key type ._llcrypt unregistered [ 2343.143437] Key type ._llcrypt registered [ 2343.149791] Key type .llcrypt registered [ 2343.283188] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2344.442583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.540600] alg: No test for adler32 (adler32-zlib) [ 2345.795647] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2346.033830] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2347.791950] Key type lgssc registered [ 2349.337473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2358.526351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2358.536701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.105339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.976343] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2369.519180] LustreError: 75010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.539328] LustreError: 75010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2373.589320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.061170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2378.851913] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2386.871387] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2393.435818] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2400.068300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2400.412346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.544350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2406.260333] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2413.509715] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2422.887161] Lustre: Failing over lustre-MDT0000 [ 2423.198200] Lustre: server umount lustre-MDT0000 complete [ 2424.289827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.303653] LustreError: 75009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.307725] Lustre: Skipped 1 previous similar message [ 2425.889017] LustreError: 75005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.908143] LustreError: 75005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2429.414653] LustreError: 75737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.447475] LustreError: 75737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2432.545705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.683634] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.864311] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2432.981424] LustreError: 78096:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2432.989500] LustreError: 78096:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.152@tcp: cfg command failed: rc = -9 [ 2433.010838] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2433.010838] [ 2433.019861] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2433.045297] LustreError: 78084:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2433.060497] LustreError: 78084:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2433.069861] LustreError: 78084:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2433.077806] LustreError: 78084:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2433.112778] Lustre: server umount lustre-MDT0000 complete [ 2433.125391] LustreError: 78084:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2434.530723] LustreError: 75010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.556875] LustreError: 75010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2439.605325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.691693] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.700422] LustreError: 75001:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2439.857124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.892789] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2441.219814] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2444.363489] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2445.298976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2445.347177] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2445.404842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2455.008578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.021292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.035511] Lustre: Skipped 1 previous similar message [ 2455.040524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.548186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.609702] Lustre: server umount lustre-OST0000 complete [ 2465.764163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2465.768086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2465.771574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2465.781916] Lustre: Skipped 2 previous similar messages [ 2468.859461] Lustre: server umount lustre-MDT0000 complete [ 2470.881318] LustreError: 75010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.905517] LustreError: 75010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2472.439384] LustreError: 74989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584804 with bad export cookie 17565766399034079870 [ 2472.451397] LustreError: 74989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2472.451821] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.763871] Lustre: server umount lustre-MDT0001 complete [ 2484.435623] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2486.932763] Key type lgssc unregistered [ 2487.211675] LNet: 79880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2487.216414] LNetError: 79880:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.240967] LNet: Removed LNI 192.168.203.152@tcp [ 2487.988485] Key type .llcrypt unregistered [ 2487.992391] Key type ._llcrypt unregistered [ 2502.577435] Key type ._llcrypt registered [ 2502.580288] Key type .llcrypt registered [ 2502.752604] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2503.033310] Key type .llcrypt unregistered [ 2503.039807] Key type ._llcrypt unregistered [ 2529.405827] Key type ._llcrypt registered [ 2529.408787] Key type .llcrypt registered [ 2529.565309] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2530.880779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.916405] alg: No test for adler32 (adler32-zlib) [ 2531.954566] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2532.168292] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2533.849056] Key type lgssc registered [ 2534.773543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2542.453842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2542.485261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.089855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.591763] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2553.315947] LustreError: 81794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.337394] LustreError: 81794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2557.141317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.479386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2561.064409] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2567.482551] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2572.558319] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2579.424401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.830201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.743404] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2589.053982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2593.531658] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2604.517161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.518828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.549646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.574834] Lustre: Skipped 1 previous similar message [ 2609.640569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2609.651293] Lustre: Skipped 2 previous similar messages [ 2610.402294] Lustre: server umount lustre-OST0000 complete [ 2614.755503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.762108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.784283] Lustre: Skipped 1 previous similar message [ 2614.798403] Lustre: Skipped 1 previous similar message [ 2619.873569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2619.885804] Lustre: Skipped 1 previous similar message [ 2620.192595] Lustre: server umount lustre-MDT0000 complete [ 2623.911895] LustreError: 81776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584955 with bad export cookie 6601838816175917466 [ 2623.915279] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2623.932218] LustreError: 81776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2624.401189] Lustre: server umount lustre-MDT0001 complete [ 2635.721637] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2639.036839] Key type lgssc unregistered [ 2639.407756] LNet: 85331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2639.422843] LNetError: 85331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2639.446150] LNet: Removed LNI 192.168.203.152@tcp [ 2640.323274] Key type .llcrypt unregistered [ 2640.326152] Key type ._llcrypt unregistered [ 2665.636489] Key type ._llcrypt registered [ 2665.638983] Key type .llcrypt registered [ 2665.721757] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2666.617497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2666.654477] alg: No test for adler32 (adler32-zlib) [ 2667.756464] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2668.031983] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2669.743274] Key type lgssc registered [ 2671.047163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2678.546933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2678.559468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.075314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.666888] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2689.508230] LustreError: 86747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.544018] LustreError: 86747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2692.825174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2693.281738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.910385] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2706.426876] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2712.493160] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2719.525623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2719.769628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.907572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2725.443237] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2733.281742] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2743.268152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.273022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.276483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2743.307543] Lustre: Skipped 1 previous similar message [ 2745.054298] Lustre: server umount lustre-OST0000 complete [ 2749.920902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.926982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.954161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.957476] Lustre: Skipped 1 previous similar message [ 2753.505929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.506545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.627559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.640347] Lustre: Skipped 2 previous similar messages [ 2763.231588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2763.478059] Lustre: server umount lustre-MDT0000 complete [ 2763.745697] LustreError: 87476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.618229] LustreError: 86730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585099 with bad export cookie 14947836720100655921 [ 2767.627972] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.961359] Lustre: server umount lustre-MDT0001 complete [ 2781.784568] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2784.822263] Key type lgssc unregistered [ 2785.211686] LNet: 90282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2785.228336] LNetError: 90282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2785.271180] LNet: Removed LNI 192.168.203.152@tcp [ 2786.356576] Key type .llcrypt unregistered [ 2786.361844] Key type ._llcrypt unregistered [ 2814.539383] Key type ._llcrypt registered [ 2814.542420] Key type .llcrypt registered [ 2814.678846] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2815.764563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2816.033794] alg: No test for adler32 (adler32-zlib) [ 2817.110801] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2817.333445] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2819.055160] Key type lgssc registered [ 2820.190580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2827.527474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2827.542502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.061622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.657601] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2838.499067] LustreError: 91696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.540846] LustreError: 91696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2843.033325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.320050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2848.103605] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2856.303253] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2862.434363] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2870.075475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2870.645112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.719455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2876.976068] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 2885.361475] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2904.546942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.568817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.585759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.597027] Lustre: Skipped 1 previous similar message [ 2909.147807] Lustre: server umount lustre-OST0000 complete [ 2914.786242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2914.801310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.802595] Lustre: Skipped 2 previous similar messages [ 2914.818622] Lustre: Skipped 1 previous similar message [ 2919.433398] Lustre: server umount lustre-MDT0000 complete [ 2919.906421] LustreError: 91691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.922626] LustreError: 91691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2923.159404] LustreError: 91678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585255 with bad export cookie 541100667590484370 [ 2923.163103] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.172050] LustreError: 91678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2923.408585] Lustre: server umount lustre-MDT0001 complete [ 2936.390346] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 2939.285702] Key type lgssc unregistered [ 2939.630376] LNet: 95251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2939.646406] LNetError: 95251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2940.714841] LNet: Removed LNI 192.168.203.152@tcp [ 2941.688795] Key type .llcrypt unregistered [ 2941.698332] Key type ._llcrypt unregistered [ 2969.585959] Key type ._llcrypt registered [ 2969.589535] Key type .llcrypt registered [ 2969.716936] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 2970.910353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2970.936519] alg: No test for adler32 (adler32-zlib) [ 2971.999789] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2972.231606] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 2973.879194] Key type lgssc registered [ 2975.061881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2982.985361] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2983.029038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2998.687300] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3003.807195] LustreError: 96638:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 3030.495564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3037.518582] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3045.207868] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3050.460938] Lustre: server umount lustre-OST0000 complete [ 3059.046891] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 00:49:48 (1781585388) [ 3067.498733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.194447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3072.829135] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3073.525033] LustreError: 98279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.562484] LustreError: 98279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3078.630093] LustreError: 98280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.747075] LustreError: 98279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.843612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3084.285285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3088.653873] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3096.622472] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3103.013308] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3110.395927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.811584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.935138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3117.436950] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3125.629546] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3144.673317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3144.684675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.690052] LustreError: Skipped 1 previous similar message [ 3144.690736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3144.729102] Lustre: Skipped 1 previous similar message [ 3148.032342] Lustre: server umount lustre-OST0000 complete [ 3154.919706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.945740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.958902] Lustre: Skipped 1 previous similar message [ 3158.831291] Lustre: server umount lustre-MDT0000 complete [ 3160.040431] LustreError: 98280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.220440] LustreError: 98259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585495 with bad export cookie 17607473229145236274 [ 3163.234510] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.243572] LustreError: 98259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3163.770934] Lustre: server umount lustre-MDT0001 complete [ 3176.991697] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3180.076779] Key type lgssc unregistered [ 3180.452549] LNet: 101971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.467490] LNetError: 101971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3181.541528] LNet: Removed LNI 192.168.203.152@tcp [ 3182.362904] Key type .llcrypt unregistered [ 3182.366460] Key type ._llcrypt unregistered [ 3232.775293] Key type ._llcrypt registered [ 3232.778488] Key type .llcrypt registered [ 3232.904440] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3234.220329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3234.237539] alg: No test for adler32 (adler32-zlib) [ 3235.407349] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3235.577922] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3237.351170] Key type lgssc registered [ 3238.850048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3247.046355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3247.080797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.771288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.791604] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3258.339631] LustreError: 104157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.366290] LustreError: 104157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3263.461674] LustreError: 104158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.041121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.503227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3269.436393] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3277.560309] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3283.514698] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3291.747772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3292.265164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.326171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3299.535464] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3308.452394] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3321.316837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.318699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.331587] Lustre: Skipped 1 previous similar message [ 3321.361371] Lustre: Skipped 1 previous similar message [ 3322.277944] Lustre: server umount lustre-OST0000 complete [ 3331.044317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3331.054925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.082740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.030634] Lustre: server umount lustre-MDT0000 complete [ 3336.673885] LustreError: 104154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.717583] LustreError: 104154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3337.191603] LustreError: 104139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585669 with bad export cookie 1427795730595679752 [ 3337.193962] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3337.205602] LustreError: 104139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3337.853188] Lustre: server umount lustre-MDT0001 complete [ 3352.192145] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3354.999611] Key type lgssc unregistered [ 3355.303696] LNet: 107694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3355.316996] LNetError: 107694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3355.342580] LNet: Removed LNI 192.168.203.152@tcp [ 3356.252969] Key type .llcrypt unregistered [ 3356.255242] Key type ._llcrypt unregistered [ 3375.905967] Key type ._llcrypt registered [ 3375.912051] Key type .llcrypt registered [ 3376.033332] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3376.928111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3376.971777] alg: No test for adler32 (adler32-zlib) [ 3378.071693] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3378.321825] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3379.975465] Key type lgssc registered [ 3381.244356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3388.924968] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3388.945202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3404.511338] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3409.631295] LustreError: 108938:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 3436.319172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.165931] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3449.474926] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3455.778320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.093442] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3456.102384] LustreError: 109961:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3456.107865] LustreError: 109961:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3456.112821] LustreError: 109961:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3456.170847] Lustre: server umount lustre-MDT0000 complete [ 3456.173835] LustreError: 109961:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3477.474857] Lustre: 108275:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781585793/real 1781585793] req@00000000b51a0f56 x1868128017452416/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781585809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3477.512289] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.522865] LustreError: 108950:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3498.115036] Lustre: server umount lustre-OST0000 complete [ 3508.119627] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3517.418754] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3556.372466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.680749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3556.734934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3556.829739] Lustre: lustre-MDT0000: new disk, initializing [ 3556.893676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.910210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3560.929049] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3571.071672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.223446] Lustre: 112755:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3571.310557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3571.312829] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3571.325137] Lustre: Skipped 1 previous similar message [ 3571.427597] Lustre: lustre-MDT0001: new disk, initializing [ 3571.549503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3571.593270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3571.602047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3576.370960] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3585.844240] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3591.294250] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3598.404739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3598.689828] Lustre: lustre-OST0000: new disk, initializing [ 3598.694508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3598.704894] Lustre: 114726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3598.784394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.905183] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3608.196537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3608.204871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3608.360671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3614.868823] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3619.897302] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3620.161591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3624.946581] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3625.155726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3629.026465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.029882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3629.041841] Lustre: Skipped 1 previous similar message [ 3629.043854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.073476] Lustre: Skipped 1 previous similar message [ 3633.480904] Lustre: server umount lustre-OST0000 complete [ 3638.245044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3638.262620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.278671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.278181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.617553] Lustre: server umount lustre-MDT0000 complete [ 3647.144072] LustreError: 112748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585979 with bad export cookie 857281145927126793 [ 3647.147816] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3647.155178] LustreError: 112748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3647.494652] Lustre: server umount lustre-MDT0001 complete [ 3655.299422] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 00:59:45 (1781585985) [ 3658.818180] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3674.048960] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3682.895333] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3714.418658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.655527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3714.684292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3714.762145] Lustre: lustre-MDT0000: new disk, initializing [ 3714.887389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.903734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3718.689398] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3728.593032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.726700] Lustre: 119622:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3728.771922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3728.774895] Lustre: Skipped 1 previous similar message [ 3728.828717] Lustre: lustre-MDT0001: new disk, initializing [ 3728.891859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.922236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3728.936126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3733.439797] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3742.936839] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3748.423046] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3755.704522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.109268] Lustre: lustre-OST0000: new disk, initializing [ 3756.114278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3756.120624] Lustre: 121601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3756.214337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.896258] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3763.302713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3763.312946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3763.455072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3772.352714] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3777.000839] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3777.182863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3781.317219] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3781.525110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3784.165503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.169777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.171525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3784.171534] Lustre: Skipped 2 previous similar messages [ 3784.180650] Lustre: Skipped 1 previous similar message [ 3784.185239] LustreError: Skipped 1 previous similar message [ 3789.289428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.293958] Lustre: Skipped 2 previous similar messages [ 3789.664851] Lustre: server umount lustre-OST0000 complete [ 3794.403605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.412192] Lustre: Skipped 1 previous similar message [ 3799.521495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.526847] Lustre: Skipped 3 previous similar messages [ 3807.201112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3807.342892] Lustre: server umount lustre-MDT0000 complete [ 3809.760888] LustreError: 120525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.784821] LustreError: 120525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3810.526042] LustreError: 119616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586142 with bad export cookie 857281145927127675 [ 3810.530065] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3810.533114] LustreError: 119616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3810.794861] Lustre: server umount lustre-MDT0001 complete [ 3817.618376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.891616] LustreError: 123819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.968193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.264840] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3823.073949] LustreError: 123820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.171119] LustreError: 123819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.271966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.596070] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3836.724728] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3841.855539] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3847.239876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.613755] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3856.683492] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3867.474184] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3873.253589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.259569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.305546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3873.316400] Lustre: Skipped 3 previous similar messages [ 3877.623419] Lustre: server umount lustre-OST0000 complete [ 3887.885776] Lustre: server umount lustre-MDT0000 complete [ 3888.608543] LustreError: 123815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.620343] LustreError: 123815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3891.395155] LustreError: 123799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586223 with bad export cookie 857281145927128557 [ 3891.412158] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3891.706483] Lustre: server umount lustre-MDT0001 complete [ 3902.119712] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 3904.342307] Key type lgssc unregistered [ 3904.548685] LNet: 127448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3904.552553] LNetError: 127448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3904.561529] LNet: Removed LNI 192.168.203.152@tcp [ 3905.109776] Key type .llcrypt unregistered [ 3905.111722] Key type ._llcrypt unregistered [ 3919.742776] Key type ._llcrypt registered [ 3919.744955] Key type .llcrypt registered [ 3919.885516] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 3934.614990] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 3935.459569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3935.481919] alg: No test for adler32 (adler32-zlib) [ 3936.595781] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 3936.794652] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 3938.479179] Key type lgssc registered [ 3939.502669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3972.138529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3972.172730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.451730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3973.482233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3973.540425] Lustre: lustre-MDT0000: new disk, initializing [ 3973.654048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.686083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3978.444830] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 3989.937371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.082353] Lustre: 130604:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3990.152252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3990.155256] Lustre: Skipped 1 previous similar message [ 3990.215687] Lustre: lustre-MDT0001: new disk, initializing [ 3990.372597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3990.416402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3990.448585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3995.651534] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4005.732303] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4011.018802] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4016.792684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.092341] Lustre: lustre-OST0000: new disk, initializing [ 4017.095454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4017.103769] Lustre: 132569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4017.175395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.955641] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4027.432171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4027.453310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4027.591876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4030.627401] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4034.699557] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4034.901691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4038.481876] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4038.688622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4043.233217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4043.247108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.250070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4043.262542] Lustre: Skipped 1 previous similar message [ 4043.284701] Lustre: Skipped 1 previous similar message [ 4046.686377] Lustre: server umount lustre-OST0000 complete [ 4051.939284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4051.943942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.953157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.478556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.483498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.912399] Lustre: server umount lustre-MDT0000 complete [ 4058.856355] LustreError: 130595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586390 with bad export cookie 4027065596016124279 [ 4058.857098] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4058.864691] LustreError: 130595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4059.129819] Lustre: server umount lustre-MDT0001 complete [ 4066.851513] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:06:36 (1781586396) [ 4074.011075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.534559] LustreError: 134980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.557843] LustreError: 134980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4074.632610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.802659] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4080.099097] LustreError: 134981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.194429] LustreError: 134980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.605153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4087.012539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.412529] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4099.346589] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4105.355162] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.746482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4107.755183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4107.789254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4107.793947] Lustre: Skipped 1 previous similar message [ 4111.844089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4111.848618] Lustre: Skipped 3 previous similar messages [ 4113.836744] Lustre: server umount lustre-MDT0000 complete [ 4116.963180] LustreError: 134975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.979249] LustreError: 134975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4117.227826] LustreError: 134961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586449 with bad export cookie 4027065596016125161 [ 4117.232095] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4117.235284] LustreError: 134961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4117.521298] Lustre: server umount lustre-MDT0001 complete [ 4125.046812] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 01:07:35 (1781586455) [ 4130.956932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.527840] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4151.647671] LustreError: 137394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4177.376625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.241951] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4190.244544] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4192.455890] Lustre: server umount lustre-OST0000 complete [ 4200.687266] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 01:08:50 (1781586530) [ 4207.372308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.779810] LustreError: 138838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.903076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.839395] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4217.315875] LustreError: 138839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.338324] LustreError: 138839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4219.711738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.524488] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4231.281204] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4237.256285] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4247.205832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4247.756400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.759459] Lustre: Skipped 1 previous similar message [ 4255.800666] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4264.670623] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4274.659668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.663357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.673022] Lustre: Skipped 3 previous similar messages [ 4274.696219] Lustre: Skipped 2 previous similar messages [ 4279.106402] Lustre: server umount lustre-MDT0000 complete [ 4279.778109] LustreError: 138838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.794491] LustreError: 138838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4282.599882] LustreError: 138819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586614 with bad export cookie 4027065596016125826 [ 4282.600674] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.607373] LustreError: 138819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4282.885783] Lustre: server umount lustre-MDT0001 complete [ 4295.315956] Lustre: server umount lustre-OST0000 complete [ 4303.584549] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 01:10:33 (1781586633) [ 4309.681597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.350773] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4315.616718] LustreError: 142496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.632263] LustreError: 142496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4321.974680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.437870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.443084] Lustre: Skipped 1 previous similar message [ 4327.624481] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4336.773028] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4343.809522] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4350.996337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.541831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4357.031946] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4363.547781] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4367.965319] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4368.170312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4373.832241] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4374.025122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4377.056909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4377.067733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.086664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4381.949637] Lustre: server umount lustre-OST0000 complete [ 4391.772248] Lustre: server umount lustre-MDT0000 complete [ 4393.444088] LustreError: 142495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.463141] LustreError: 142495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4395.722379] LustreError: 142476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586727 with bad export cookie 4027065596016126897 [ 4395.725538] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.736401] LustreError: 142476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4396.173346] Lustre: server umount lustre-MDT0001 complete [ 4404.686375] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 01:12:14 (1781586734) [ 4411.132283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.783629] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4431.903459] LustreError: 146653:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4457.697791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.714553] Lustre: Skipped 1 previous similar message [ 4464.073409] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4471.331432] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4478.197075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.860509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4483.417301] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4492.185728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.261589] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4503.672988] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4509.364724] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4514.085788] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4514.308646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4518.183588] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4518.425890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4523.504514] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.505340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4523.507990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4523.508016] Lustre: Skipped 4 previous similar messages [ 4523.519642] Lustre: Skipped 5 previous similar messages [ 4523.525731] LustreError: Skipped 1 previous similar message [ 4526.465303] Lustre: server umount lustre-MDT0000 complete [ 4528.608500] LustreError: 147691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.623624] LustreError: 147691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4529.974095] LustreError: 148437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586861 with bad export cookie 4027065596016127919 [ 4529.983117] LustreError: 148437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4529.996015] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4530.270864] Lustre: server umount lustre-MDT0001 complete [ 4544.142840] Lustre: server umount lustre-OST0000 complete [ 4551.784785] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 01:14:41 (1781586881) [ 4557.637773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.215639] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4578.336235] LustreError: 150852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 4609.688162] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4617.532786] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4623.343596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.850265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4628.266788] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4636.324292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.854124] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4649.337521] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4656.271699] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4664.122743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4664.277940] Lustre: lustre-OST0001: new disk, initializing [ 4664.281358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4664.286655] Lustre: 153724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4671.085140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4671.097468] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4671.204453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4671.956699] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4681.415537] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4686.072171] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4686.241567] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4690.653967] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4690.929983] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4695.524447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4695.542480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.602942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.606621] Lustre: Skipped 2 previous similar messages [ 4707.295194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4707.391354] Lustre: server umount lustre-OST0000 complete [ 4717.234853] Lustre: server umount lustre-OST0001 complete [ 4727.215140] Lustre: server umount lustre-MDT0000 complete [ 4731.200774] LustreError: 150860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587063 with bad export cookie 4027065596016128871 [ 4731.206460] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4731.216376] LustreError: 150860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4731.590777] Lustre: server umount lustre-MDT0001 complete [ 4754.261694] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 01:18:04 (1781587084) [ 4756.144711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4757.675668] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4759.371789] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 01:18:09 (1781587089) [ 4766.080647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4766.309806] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4766.364658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4766.648427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.656637] Lustre: Skipped 6 previous similar messages [ 4771.080533] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4780.710349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.896132] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4780.947667] Lustre: 157283:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4786.552579] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4794.726772] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4799.655831] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4805.527263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.750634] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4812.555483] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4816.050870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4822.769211] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4829.642695] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4829.786290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4833.710658] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4834.015778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4836.854361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.854961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.857148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.857155] Lustre: Skipped 10 previous similar messages [ 4836.863622] Lustre: Skipped 6 previous similar messages [ 4841.838393] Lustre: server umount lustre-OST0000 complete [ 4862.671865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4868.949395] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4876.204565] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.224126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.238761] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4876.321994] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4881.476889] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4881.757265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4886.771865] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4887.009639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4893.627169] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c44c49be800.ost_server_uuid 50 [ 4895.700595] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c44c49be800.ost_server_uuid in FULL state after 0 sec [ 4895.875134] LustreError: 157290:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.52@tcp) returned error from blocking AST (req@00000000e1bcd1c9 x1868128602816256 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000cfc6c008/0x37e302d2c1bcfcc1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.52@tcp remote: 0xa331138a4150dd78 expref: 5 pid: 158425 timeout: 4995 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4895.928540] LustreError: lustre-MDT0000: A client on nid 192.168.203.52@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4895.954084] LustreError: 157279:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.52@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000cfc6c008/0x37e302d2c1bcfcc1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.52@tcp remote: 0xa331138a4150dd78 expref: 6 pid: 158425 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4901.858933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4901.892593] LustreError: Skipped 2 previous similar messages [ 4906.375926] Lustre: server umount lustre-OST0000 complete [ 4917.584192] Lustre: server umount lustre-MDT0000 complete [ 4921.315610] LustreError: 157290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.344782] LustreError: 157290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4921.364841] LustreError: 157274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587253 with bad export cookie 4027065596016130054 [ 4921.372751] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4921.374369] LustreError: 157274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4927.887576] Lustre: server umount lustre-MDT0001 complete [ 4938.153197] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 4940.578688] Key type lgssc unregistered [ 4940.854699] LNet: 163395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4940.863125] LNetError: 163395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4940.892206] LNet: Removed LNI 192.168.203.152@tcp [ 4941.741250] Key type .llcrypt unregistered [ 4941.747916] Key type ._llcrypt unregistered [ 4963.889920] Key type ._llcrypt registered [ 4963.892303] Key type .llcrypt registered [ 4963.994511] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 4965.059758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4965.175171] alg: No test for adler32 (adler32-zlib) [ 4966.183176] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 4966.436848] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 4968.119112] Key type lgssc registered [ 4969.325636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4976.915934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4976.938589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.441068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.576709] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 4988.896483] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.917653] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4992.655300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.054549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4997.734254] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5005.721281] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5012.055402] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5019.046831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5019.411201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.489854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5025.521708] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5033.068915] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5042.175137] Lustre: Failing over lustre-MDT0000 [ 5042.389483] Lustre: server umount lustre-MDT0000 complete [ 5043.173871] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.179357] LustreError: 164811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.191607] Lustre: Skipped 2 previous similar messages [ 5043.219731] LustreError: 164811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5048.291234] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5048.302591] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5053.409207] LustreError: 165540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.430096] LustreError: 165540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5054.158345] LustreError: 164791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.52@tcp arrived at 1781587386 with bad export cookie 3964145417567473181 [ 5054.181665] LustreError: 164791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5058.530515] LustreError: 164811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5058.550533] LustreError: 164811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5059.553593] Lustre: 164126:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781587375/real 1781587375] req@000000002ca06bed x1868129682613888/t0(0) o400->MGC192.168.203.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781587391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5059.588722] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.787914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.791662] LustreError: 164122:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003f7ee603 x1868129682620416/t0(0) o250->MGC192.168.203.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5069.815890] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.844174] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5070.139454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.214169] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5074.370558] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5075.431184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5075.447407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5076.537080] Lustre: lustre-MDT0000: Denying connection for new client 80ea2d5c-6536-407d-9b49-4c85c9ef6bb0 (at 192.168.203.52@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5085.666017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.675083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.693155] Lustre: Skipped 1 previous similar message [ 5090.787257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.793169] Lustre: Skipped 2 previous similar messages [ 5091.108284] Lustre: server umount lustre-OST0000 complete [ 5095.424253] LustreError: 168544:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5095.437522] Lustre: 167893:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5095.443534] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5095.457051] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5095.469817] LustreError: 167893:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000014067c9b x1868129682635904/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5095.492964] LustreError: 167893:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5095.504299] LustreError: 167893:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5095.519670] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5095.834880] Lustre: server umount lustre-MDT0000 complete [ 5095.906131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.918605] LustreError: 165983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5099.778380] LustreError: 164793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587431 with bad export cookie 3964145417567473286 [ 5099.804925] LustreError: 164793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5099.807184] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.120638] Lustre: server umount lustre-MDT0001 complete [ 5113.580610] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5116.778408] Key type lgssc unregistered [ 5117.136125] LNet: 169297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5117.147083] LNetError: 169297:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5117.165798] LNet: Removed LNI 192.168.203.152@tcp [ 5118.007721] Key type .llcrypt unregistered [ 5118.009974] Key type ._llcrypt unregistered [ 5145.132266] Key type ._llcrypt registered [ 5145.136863] Key type .llcrypt registered [ 5145.287927] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5146.355170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5146.447078] alg: No test for adler32 (adler32-zlib) [ 5147.718501] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5147.991131] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5149.640793] Key type lgssc registered [ 5150.826756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5160.183043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5160.213188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.912402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5166.940902] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5171.169506] LustreError: 170711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.187336] LustreError: 170711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5176.286993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.290264] LustreError: 170712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.732971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5181.437989] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5189.440201] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5195.582958] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5202.114524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5202.370809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5206.530352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5209.128172] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5217.131762] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5226.984581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5226.984916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.986602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5226.992974] LustreError: Skipped 1 previous similar message [ 5227.011384] Lustre: Skipped 1 previous similar message [ 5229.909218] Lustre: server umount lustre-OST0000 complete [ 5237.218847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.229876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5237.262048] Lustre: Skipped 1 previous similar message [ 5237.285844] Lustre: Skipped 2 previous similar messages [ 5240.297142] Lustre: server umount lustre-MDT0000 complete [ 5242.344920] LustreError: 171718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.378302] LustreError: 171718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5244.873079] LustreError: 173493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587576 with bad export cookie 8083639007903093728 [ 5244.887659] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.889977] LustreError: 173493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5245.536928] Lustre: server umount lustre-MDT0001 complete [ 5257.229226] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5260.434371] Key type lgssc unregistered [ 5260.727587] LNet: 174244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5260.737662] LNetError: 174244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5260.756607] LNet: Removed LNI 192.168.203.152@tcp [ 5261.631955] Key type .llcrypt unregistered [ 5261.633927] Key type ._llcrypt unregistered [ 5290.683066] Key type ._llcrypt registered [ 5290.687633] Key type .llcrypt registered [ 5290.784912] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5291.507080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5291.527836] alg: No test for adler32 (adler32-zlib) [ 5292.540715] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5292.762678] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5294.447678] Key type lgssc registered [ 5295.502802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5302.085304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5302.097513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.446982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.365227] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5314.026866] LustreError: 175925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5314.054380] LustreError: 175925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5315.644504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.031574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5320.956346] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5328.562224] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5335.421197] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5344.601951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.035539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.154190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5351.750784] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5359.918695] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5368.807222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.824137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.845574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5370.851155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5370.852169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.871213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.350323] Lustre: server umount lustre-OST0000 complete [ 5377.511631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5377.529724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.559435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5377.569577] Lustre: Skipped 1 previous similar message [ 5381.089389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5381.100984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5383.420903] Lustre: server umount lustre-MDT0000 complete [ 5386.215297] LustreError: 176920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.239307] LustreError: 176920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5387.469938] LustreError: 175907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587719 with bad export cookie 14581116522597542991 [ 5387.484089] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.490191] LustreError: 175907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5387.981635] Lustre: server umount lustre-MDT0001 complete [ 5402.720211] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5405.970674] Key type lgssc unregistered [ 5406.334445] LNet: 179459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5406.346906] LNetError: 179459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5406.391784] LNet: Removed LNI 192.168.203.152@tcp [ 5407.318359] Key type .llcrypt unregistered [ 5407.320317] Key type ._llcrypt unregistered [ 5431.270903] Key type ._llcrypt registered [ 5431.282916] Key type .llcrypt registered [ 5431.420840] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5432.907592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5432.979127] alg: No test for adler32 (adler32-zlib) [ 5434.109882] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5434.351055] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5436.039316] Key type lgssc registered [ 5437.194203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5448.298946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5448.366259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.828674] Lustre: *** cfs_fail_loc=135, val=0*** [ 5449.959126] LustreError: 180984:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5449.969507] LustreError: 180984:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.152@tcp: cfg command failed: rc = -2 [ 5449.977828] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5449.977828] [ 5449.990373] LustreError: MGC192.168.203.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5450.004969] LustreError: 180945:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5450.020447] LustreError: 180945:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5450.027901] LustreError: 180945:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5450.037773] LustreError: 180945:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5450.125509] Lustre: server umount lustre-MDT0000 complete [ 5450.128487] LustreError: 180945:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5457.503418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.982735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.804820] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5467.107128] LustreError: 181355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.134885] LustreError: 181355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5472.229207] LustreError: 181356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5472.781535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5473.110221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5477.657642] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5484.686036] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5490.440547] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5492.521405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.535179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5492.543451] Lustre: Skipped 1 previous similar message [ 5497.698876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5497.707038] Lustre: Skipped 1 previous similar message [ 5498.786447] Lustre: server umount lustre-MDT0000 complete [ 5502.184101] LustreError: 182092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781587834 with bad export cookie 15060074500722204544 [ 5502.193049] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5502.195374] LustreError: 182092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5502.647856] Lustre: server umount lustre-MDT0001 complete [ 5513.560782] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5516.358773] Key type lgssc unregistered [ 5516.701436] LNet: 183678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5516.713758] LNetError: 183678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5516.750475] LNet: Removed LNI 192.168.203.152@tcp [ 5517.592160] Key type .llcrypt unregistered [ 5517.594415] Key type ._llcrypt unregistered [ 5539.184524] Key type ._llcrypt registered [ 5539.189829] Key type .llcrypt registered [ 5539.308707] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5539.633399] Key type .llcrypt unregistered [ 5539.635225] Key type ._llcrypt unregistered [ 5556.640466] Key type ._llcrypt registered [ 5556.644172] Key type .llcrypt registered [ 5556.756757] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5557.804549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5557.916091] alg: No test for adler32 (adler32-zlib) [ 5559.057667] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5559.257737] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5560.975533] Key type lgssc registered [ 5562.505205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5570.372507] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5570.422304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5586.081505] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5591.201559] LustreError: 185809:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.152@tcp: failed processing log, type 4: rc = -110 [ 5617.952388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.646677] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5632.812716] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5640.786779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.740240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.812908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5646.602647] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5646.823243] LustreError: 185835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.854940] LustreError: 185835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5651.937531] LustreError: 185834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.964770] LustreError: 186846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.004493] LustreError: 186846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5655.228422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.655293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5660.017142] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5668.653232] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5675.463517] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5678.687266] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5691.367961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5691.368955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.382383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5691.394061] Lustre: Skipped 1 previous similar message [ 5691.404765] Lustre: Skipped 1 previous similar message [ 5696.492909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5696.503969] Lustre: Skipped 1 previous similar message [ 5701.626708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.632116] Lustre: Skipped 1 previous similar message [ 5702.111288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5702.334817] Lustre: server umount lustre-OST0000 complete [ 5706.724899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5706.732898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5706.754941] Lustre: Skipped 1 previous similar message [ 5706.771244] Lustre: Skipped 1 previous similar message [ 5711.845754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.864377] Lustre: Skipped 1 previous similar message [ 5712.501481] Lustre: server umount lustre-MDT0000 complete [ 5716.615952] LustreError: 185817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588048 with bad export cookie 10966011264663813212 [ 5716.617032] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5716.627763] LustreError: 185817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5717.126778] Lustre: server umount lustre-MDT0001 complete [ 5728.309220] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5731.451883] Key type lgssc unregistered [ 5731.781743] LNet: 189729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5731.793949] LNetError: 189729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5731.807773] LNet: Removed LNI 192.168.203.152@tcp [ 5732.672160] Key type .llcrypt unregistered [ 5732.676222] Key type ._llcrypt unregistered [ 5757.123570] Key type ._llcrypt registered [ 5757.126198] Key type .llcrypt registered [ 5757.284245] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5758.535059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5758.736096] alg: No test for adler32 (adler32-zlib) [ 5759.784829] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5760.029988] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5761.752086] Key type lgssc registered [ 5763.097301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5771.161167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5771.215448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5772.883949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.059224] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5783.022073] LustreError: 191148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.050372] LustreError: 191148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5788.151936] LustreError: 191149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.288047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.683978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5793.414387] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5801.941674] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5807.758832] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5815.083968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5815.511444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5817.614327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5821.347441] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5829.531693] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5834.901752] Lustre: Failing over lustre-MDT0000 [ 5835.187960] Lustre: server umount lustre-MDT0000 complete [ 5837.351194] LustreError: 191144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.331740] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.344080] Lustre: Skipped 2 previous similar messages [ 5839.352794] LustreError: 191143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.381508] LustreError: 191143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5844.451392] LustreError: 192992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5844.487112] LustreError: 192992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5852.676706] LustreError: 191143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.695711] LustreError: 191143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5854.382845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.616928] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5855.010312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.076833] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5857.735199] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5860.347855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5860.371934] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5860.444825] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5860.448911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5865.461571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5871.443262] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5880.801118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.822221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5883.023379] Lustre: server umount lustre-OST0000 complete [ 5891.041202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.044139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5891.066191] Lustre: Skipped 1 previous similar message [ 5891.075072] Lustre: Skipped 1 previous similar message [ 5893.903099] Lustre: server umount lustre-MDT0000 complete [ 5896.160716] LustreError: 192149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5896.184132] LustreError: 192149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5897.920075] LustreError: 191128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588229 with bad export cookie 10982632185874574038 [ 5897.927478] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5897.933600] LustreError: 191128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5907.762787] Lustre: server umount lustre-MDT0001 complete [ 5920.037847] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 5922.480675] Key type lgssc unregistered [ 5922.797504] LNet: 195989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5922.804361] LNetError: 195989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5922.819534] LNet: Removed LNI 192.168.203.152@tcp [ 5923.739440] Key type .llcrypt unregistered [ 5923.745900] Key type ._llcrypt unregistered [ 5949.410865] Key type ._llcrypt registered [ 5949.414952] Key type .llcrypt registered [ 5949.570680] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 5950.785411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5950.870762] alg: No test for adler32 (adler32-zlib) [ 5951.860630] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 5952.080748] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 5953.727183] Key type lgssc registered [ 5954.769360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5961.536681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5961.586372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.947745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.830885] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5972.451075] LustreError: 197404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.477162] LustreError: 197404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5974.034038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.330335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5978.650574] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 5985.529589] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5991.151274] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5998.389284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5998.876709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6005.156854] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6007.165709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6013.277107] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6017.409641] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6022.061101] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6040.631365] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6051.297868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6051.315875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.338363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6053.348956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6053.357682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6053.367027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.560716] Lustre: server umount lustre-OST0000 complete [ 6058.466016] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.476782] Lustre: Skipped 1 previous similar message [ 6058.482390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.496879] Lustre: Skipped 2 previous similar messages [ 6063.587814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6063.593608] Lustre: Skipped 1 previous similar message [ 6064.791360] Lustre: server umount lustre-MDT0000 complete [ 6068.704540] LustreError: 197400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.729293] LustreError: 197400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6069.036682] LustreError: 197385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588400 with bad export cookie 6481748110950168783 [ 6069.037737] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.046378] LustreError: 197385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6069.519358] Lustre: server umount lustre-MDT0001 complete [ 6082.511123] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6084.936812] Key type lgssc unregistered [ 6085.207110] LNet: 201272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6085.212494] LNetError: 201272:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6085.237147] LNet: Removed LNI 192.168.203.152@tcp [ 6086.121480] Key type .llcrypt unregistered [ 6086.123340] Key type ._llcrypt unregistered [ 6106.406209] Key type ._llcrypt registered [ 6106.414941] Key type .llcrypt registered [ 6106.559702] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6107.579193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6107.601856] alg: No test for adler32 (adler32-zlib) [ 6108.739820] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6109.080424] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6110.791265] Key type lgssc registered [ 6111.916320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6120.377765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6120.427508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.962135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.497212] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6131.190448] LustreError: 202554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6131.213588] LustreError: 202554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6134.824487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.355619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6140.219269] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6148.123157] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6154.339476] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6161.820230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6162.211650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6166.350455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6168.525236] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6176.808250] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6182.357851] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6191.758136] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6201.591123] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6208.219044] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6216.674704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6216.679483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.683542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6216.694971] Lustre: Skipped 1 previous similar message [ 6216.706366] Lustre: Skipped 1 previous similar message [ 6221.796712] Lustre: server umount lustre-OST0000 complete [ 6226.913697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.937414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6231.972850] Lustre: server umount lustre-MDT0000 complete [ 6232.032968] LustreError: 202549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.056833] LustreError: 202549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6236.129403] LustreError: 202553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6236.150326] LustreError: 202553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6236.181861] LustreError: 202535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588568 with bad export cookie 7873416456688767866 [ 6236.190821] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.199931] LustreError: 202535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6236.660783] Lustre: server umount lustre-MDT0001 complete [ 6247.722874] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6250.299400] Key type lgssc unregistered [ 6250.559605] LNet: 207332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6250.566972] LNetError: 207332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6250.581244] LNet: Removed LNI 192.168.203.152@tcp [ 6251.322752] Key type .llcrypt unregistered [ 6251.326655] Key type ._llcrypt unregistered [ 6274.156278] Key type ._llcrypt registered [ 6274.159719] Key type .llcrypt registered [ 6274.257754] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6275.160975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.212048] alg: No test for adler32 (adler32-zlib) [ 6276.204150] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6276.381806] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6278.049312] Key type lgssc registered [ 6279.050287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.036390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6286.062738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.747175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6291.471456] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6298.082220] LustreError: 208745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.098173] LustreError: 208745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6299.066535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6299.460272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6303.094159] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6309.566406] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6314.031772] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6320.883234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6321.221747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.761118] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6332.524955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6334.939502] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6353.380185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.384141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6353.399975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.413132] Lustre: Skipped 1 previous similar message [ 6356.711919] Lustre: server umount lustre-OST0000 complete [ 6361.058813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6361.078267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.092821] Lustre: Skipped 1 previous similar message [ 6361.101550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.616450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.616885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.640554] Lustre: Skipped 1 previous similar message [ 6366.105362] Lustre: server umount lustre-MDT0000 complete [ 6368.739525] LustreError: 208741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.755664] LustreError: 208741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6370.169645] LustreError: 208726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588702 with bad export cookie 10266650873741363557 [ 6370.171389] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6370.180714] LustreError: 208726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6370.545359] Lustre: server umount lustre-MDT0001 complete [ 6381.749523] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6384.137143] Key type lgssc unregistered [ 6384.395086] LNet: 212428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6384.400900] LNetError: 212428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6385.445212] LNet: Removed LNI 192.168.203.152@tcp [ 6386.253288] Key type .llcrypt unregistered [ 6386.257614] Key type ._llcrypt unregistered [ 6403.838109] Key type ._llcrypt registered [ 6403.840918] Key type .llcrypt registered [ 6403.935975] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6404.746634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6404.814430] alg: No test for adler32 (adler32-zlib) [ 6405.891816] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6406.086477] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6407.791172] Key type lgssc registered [ 6408.759629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6415.465487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6415.497940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.070517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6421.013655] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6426.603689] LustreError: 213692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.620301] LustreError: 213692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6429.189672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.589529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6434.139526] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6441.120408] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6446.423844] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6453.107302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.507586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6459.288839] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6461.756684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6467.401891] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6482.402467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6482.403187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.412749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6482.427776] Lustre: Skipped 1 previous similar message [ 6482.439748] Lustre: Skipped 1 previous similar message [ 6487.524874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6487.533390] Lustre: Skipped 1 previous similar message [ 6488.264971] Lustre: server umount lustre-OST0000 complete [ 6492.643809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6492.653596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6492.658031] Lustre: Skipped 1 previous similar message [ 6492.672515] Lustre: Skipped 1 previous similar message [ 6497.764091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6497.772478] Lustre: Skipped 1 previous similar message [ 6498.184432] Lustre: server umount lustre-MDT0000 complete [ 6501.941941] LustreError: 213672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588833 with bad export cookie 18266933716404346057 [ 6501.951339] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6501.956451] LustreError: 213672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6502.280107] Lustre: server umount lustre-MDT0001 complete [ 6513.303456] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6515.844811] Key type lgssc unregistered [ 6516.120822] LNet: 217377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6516.131534] LNetError: 217377:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6516.148314] LNet: Removed LNI 192.168.203.152@tcp [ 6516.796362] Key type .llcrypt unregistered [ 6516.799779] Key type ._llcrypt unregistered [ 6534.800639] Key type ._llcrypt registered [ 6534.804674] Key type .llcrypt registered [ 6534.887195] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6535.853220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6535.926618] alg: No test for adler32 (adler32-zlib) [ 6536.983545] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6537.156933] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6538.823309] Key type lgssc registered [ 6539.817830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6546.979719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6547.023730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.412956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.236147] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6558.697847] LustreError: 218644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.714388] LustreError: 218644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6563.210171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.903889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6568.640515] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6577.034049] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6584.095918] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6590.894730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6591.387220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.477793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6598.072728] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6605.839906] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6615.770749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6615.897066] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6616.262126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6622.780299] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6630.451316] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6632.191174] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6643.732157] Lustre: Permanently deactivating lustre-OST0001 [ 6643.748556] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6652.466096] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6660.307545] Lustre: Permanently reactivating lustre-OST0001 [ 6660.319789] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6660.323499] Lustre: Skipped 2 previous similar messages [ 6661.490836] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.507589] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6661.513271] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6661.542093] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6661.583023] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6673.887106] Lustre: server umount lustre-OST0001 complete [ 6674.435224] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.453681] Lustre: Skipped 1 previous similar message [ 6674.467547] LustreError: 221842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.485537] LustreError: 221842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6679.522263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.526805] LustreError: 220468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6679.550320] Lustre: Skipped 1 previous similar message [ 6679.551636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6679.575799] LustreError: 220468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6683.459050] Lustre: server umount lustre-OST0000 complete [ 6689.760143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.770127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6689.775248] Lustre: Skipped 2 previous similar messages [ 6689.793723] Lustre: Skipped 2 previous similar messages [ 6694.885498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6694.902544] Lustre: Skipped 1 previous similar message [ 6700.005053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.021437] Lustre: Skipped 1 previous similar message [ 6701.023702] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6701.208593] Lustre: server umount lustre-MDT0000 complete [ 6704.097760] LustreError: 218640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.112558] LustreError: 218640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6704.155867] LustreError: 222269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589036 with bad export cookie 532027050774074754 [ 6704.163562] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6704.165682] LustreError: 222269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6704.480260] Lustre: server umount lustre-MDT0001 complete [ 6713.848321] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 6716.037736] Key type lgssc unregistered [ 6716.312750] LNet: 223739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6716.320719] LNetError: 223739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6716.359846] LNet: Removed LNI 192.168.203.152@tcp [ 6716.946274] Key type .llcrypt unregistered [ 6716.948919] Key type ._llcrypt unregistered [ 6747.972789] Key type ._llcrypt registered [ 6747.974256] Key type .llcrypt registered [ 6748.046400] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 6748.919685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6748.940043] alg: No test for adler32 (adler32-zlib) [ 6749.998195] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 6750.213742] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 6751.911152] Key type lgssc registered [ 6752.827400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6759.496593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6759.530764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.822447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6760.856858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6761.012298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.697628] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6773.953587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.086200] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6774.109515] Lustre: 225923:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6774.338655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6778.309536] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6786.520733] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6793.187761] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6799.655251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6799.857941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6800.071311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6805.943490] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 6808.174696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6812.975587] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6815.799409] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6820.384766] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6830.224852] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6840.103848] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6857.851052] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6857.859137] Lustre: Skipped 1 previous similar message [ 6881.078920] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6881.091431] Lustre: Skipped 2 previous similar messages [ 6917.215206] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6917.220640] Lustre: Skipped 4 previous similar messages [ 6987.235488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.245204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.263154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.746726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.762445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.790676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6992.151367] Lustre: server umount lustre-OST0000 complete [ 6998.026849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.030986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6998.053114] Lustre: Skipped 1 previous similar message [ 6998.074757] Lustre: Skipped 1 previous similar message [ 7003.106813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.122956] Lustre: Skipped 1 previous similar message [ 7007.201053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7007.211703] Lustre: Skipped 1 previous similar message [ 7010.271172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7010.484383] Lustre: server umount lustre-MDT0000 complete [ 7015.193205] LustreError: 228491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589347 with bad export cookie 11596026415543408291 [ 7015.195351] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7015.203449] LustreError: 228491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7015.569336] Lustre: server umount lustre-MDT0001 complete [ 7026.493308] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7029.566990] Key type lgssc unregistered [ 7029.987818] LNet: 230709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7030.003018] LNetError: 230709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7030.021843] LNet: Removed LNI 192.168.203.152@tcp [ 7031.026785] Key type .llcrypt unregistered [ 7031.029956] Key type ._llcrypt unregistered [ 7055.634445] Key type ._llcrypt registered [ 7055.637170] Key type .llcrypt registered [ 7055.754653] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7057.090668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7057.147574] alg: No test for adler32 (adler32-zlib) [ 7058.193573] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 7058.325421] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7059.980454] Key type lgssc registered [ 7061.544058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7070.803259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7070.830093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.361370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.000278] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7082.473152] LustreError: 232124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7082.497239] LustreError: 232124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7086.063211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.412412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7090.817467] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7098.288853] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7104.404962] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7111.046618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7111.449408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.893342] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_default_debug -1 all [ 7117.622380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7123.692466] Lustre: DEBUG MARKER: oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7142.119313] Lustre: server umount lustre-OST0000 complete [ 7148.000272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7148.020945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7148.041806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7153.636141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7153.657493] Lustre: Skipped 3 previous similar messages [ 7155.754897] Lustre: server umount lustre-MDT0000 complete [ 7158.754667] LustreError: 232853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.774441] LustreError: 232853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7159.469349] LustreError: 232105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781589491 with bad export cookie 17086919779491829347 [ 7159.480141] LustreError: MGC192.168.203.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7159.486990] LustreError: 232105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7170.202941] Lustre: server umount lustre-MDT0001 complete [ 7181.396546] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7184.007439] Key type lgssc unregistered [ 7184.298706] LNet: 235815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7184.307790] LNetError: 235815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7184.333242] LNet: Removed LNI 192.168.203.152@tcp [ 7185.088647] Key type .llcrypt unregistered [ 7185.092844] Key type ._llcrypt unregistered [ 7206.613498] Key type ._llcrypt registered [ 7206.616246] Key type .llcrypt registered [ 7206.735957] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7207.717840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7207.738275] alg: No test for adler32 (adler32-zlib) [ 7208.812346] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 7209.094776] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7210.879304] Key type lgssc registered [ 7211.933575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7483.316145] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7486.136575] Key type lgssc unregistered [ 7486.417680] LNet: 237855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7486.433348] LNetError: 237855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7486.459020] LNet: Removed LNI 192.168.203.152@tcp [ 7487.246958] Key type .llcrypt unregistered [ 7487.251593] Key type ._llcrypt unregistered [ 7516.205403] Key type ._llcrypt registered [ 7516.207172] Key type .llcrypt registered [ 7516.328913] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing set_hostid [ 7533.855528] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing load_modules_local [ 7535.180916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7535.205592] alg: No test for adler32 (adler32-zlib) [ 7536.384554] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 7536.650587] LNet: Added LNI 192.168.203.152@tcp [8/256/0/180] [ 7538.359158] Key type lgssc registered [ 7539.607824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7574.120946] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7292 sec ========== 02:05:04 (1781589904) [ 7575.808749] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:05:05 (1781589905) === [ 7579.270136] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:05:09 (1781589909) === [ 7619.704399] Lustre: DEBUG MARKER: oleg352-server.virtnet: executing unload_modules_local [ 7622.297968] Key type lgssc unregistered [ 7622.591492] LNet: 243093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7622.606899] LNetError: 243093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7622.623259] LNet: Removed LNI 192.168.203.152@tcp [ 7623.465691] Key type .llcrypt unregistered [ 7623.468728] Key type ._llcrypt unregistered