[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 570478865 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002391] x2apic enabled [ 0.003015] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.006914] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007033] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009163] LSM: Security Framework initializing [ 0.010075] Yama: becoming mindful. [ 0.011055] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.021532] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026558] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028206] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] *** VALIDATE tmpfs *** [ 0.031083] *** VALIDATE proc *** [ 0.032310] *** VALIDATE cgroup *** [ 0.033024] *** VALIDATE cgroup2 *** [ 0.035006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037100] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039041] Spectre V2 : User space: Vulnerable [ 0.040017] Speculative Store Bypass: Vulnerable [ 0.043789] debug: unmapping init [mem 0xffffffff8b259000-0xffffffff8b260fff] [ 0.045828] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046813] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054377] rcu: Hierarchical SRCU implementation. [ 0.056678] smp: Bringing up secondary CPUs ... [ 0.057681] x86: Booting SMP configuration: [ 0.058040] .... node #0, CPUs: #1 #2 #3 [ 0.063133] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.274269] node 0 deferred pages initialised in 206ms [ 0.278126] devtmpfs: initialized [ 0.279276] x86/mm: Memory block size: 128MB [ 0.281662] gcov: version magic: 0x41383552 [ 0.284275] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.288330] pinctrl core: initialized pinctrl subsystem [ 0.290143] [ 0.290528] ************************************************************* [ 0.292012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294010] ** ** [ 0.296013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.297009] ** ** [ 0.299011] ** This means that this kernel is built to expose internal ** [ 0.301011] ** IOMMU data structures, which may compromise security on ** [ 0.302009] ** your system. ** [ 0.304011] ** ** [ 0.306013] ** If you see this message and you are not debugging the ** [ 0.308012] ** kernel, report this immediately to your vendor! ** [ 0.309011] ** ** [ 0.311012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.313010] ************************************************************* [ 0.314629] NET: Registered protocol family 16 [ 0.316402] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.318048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.320048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322409] cpuidle: using governor menu [ 0.323514] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.331048] PCI: Using configuration type 1 for base access [ 0.335156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.350150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.353033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.357325] cryptd: max_cpu_qlen set to 1000 [ 0.359612] ACPI: Added _OSI(Module Device) [ 0.360037] ACPI: Added _OSI(Processor Device) [ 0.361000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.362028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.368000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.374314] ACPI: Interpreter enabled [ 0.375045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.376012] ACPI: Using IOAPIC for interrupt routing [ 0.378090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.380358] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.393856] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.396030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.397017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.401079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.405161] acpiphp: Slot [2] registered [ 0.406099] acpiphp: Slot [5] registered [ 0.408160] acpiphp: Slot [6] registered [ 0.409103] acpiphp: Slot [7] registered [ 0.410082] acpiphp: Slot [8] registered [ 0.411059] acpiphp: Slot [9] registered [ 0.412115] acpiphp: Slot [10] registered [ 0.414110] acpiphp: Slot [3] registered [ 0.415091] acpiphp: Slot [4] registered [ 0.417086] acpiphp: Slot [11] registered [ 0.419128] acpiphp: Slot [12] registered [ 0.421115] acpiphp: Slot [13] registered [ 0.423277] acpiphp: Slot [14] registered [ 0.425105] acpiphp: Slot [15] registered [ 0.428207] acpiphp: Slot [16] registered [ 0.430115] acpiphp: Slot [17] registered [ 0.432114] acpiphp: Slot [18] registered [ 0.434099] acpiphp: Slot [19] registered [ 0.435253] acpiphp: Slot [20] registered [ 0.437247] acpiphp: Slot [21] registered [ 0.439134] acpiphp: Slot [22] registered [ 0.441131] acpiphp: Slot [23] registered [ 0.443131] acpiphp: Slot [24] registered [ 0.445114] acpiphp: Slot [25] registered [ 0.447115] acpiphp: Slot [26] registered [ 0.449110] acpiphp: Slot [27] registered [ 0.450104] acpiphp: Slot [28] registered [ 0.452099] acpiphp: Slot [29] registered [ 0.453078] acpiphp: Slot [30] registered [ 0.454091] acpiphp: Slot [31] registered [ 0.455065] PCI host bridge to bus 0000:00 [ 0.456015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.458020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.460027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.462023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.464020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.467049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.469225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.473196] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.476482] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.488020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.493055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.496022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.498018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.501017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.504704] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.506818] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.509112] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.512884] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.519033] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.532733] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.538016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.543679] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.550018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.557016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.574032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.585805] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.595016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.608017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.623016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.635968] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.646081] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.653020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.669021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.681920] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.688018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.694017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.710014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.720816] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.727020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.738021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.756021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.764000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.772023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.779020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.796019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.811495] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.818246] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.823472] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.825336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.827270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.834052] iommu: Default domain type: Passthrough [ 0.836536] SCSI subsystem initialized [ 0.842300] ACPI: bus type USB registered [ 0.844250] usbcore: registered new interface driver usbfs [ 0.847113] usbcore: registered new interface driver hub [ 0.848128] usbcore: registered new device driver usb [ 0.850415] pps_core: LinuxPPS API ver. 1 registered [ 0.852013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.855085] PTP clock support registered [ 0.857073] EDAC MC: Ver: 3.0.0 [ 0.860163] PCI: Using ACPI for IRQ routing [ 0.861000] NetLabel: Initializing [ 0.861000] NetLabel: domain hash size = 128 [ 0.862011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.863099] NetLabel: unlabeled traffic allowed by default [ 0.864253] vgaarb: loaded [ 0.865312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.867018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.875829] clocksource: Switched to clocksource kvm-clock [ 1.021312] VFS: Disk quotas dquot_6.6.0 [ 1.025489] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.028605] *** VALIDATE ramfs *** [ 1.032461] *** VALIDATE hugetlbfs *** [ 1.034953] pnp: PnP ACPI init [ 1.037780] pnp: PnP ACPI: found 6 devices [ 1.064632] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.068537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.071176] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.075065] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.080791] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.083428] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.086436] NET: Registered protocol family 2 [ 1.089159] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.094766] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.099083] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.105467] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.109484] TCP: Hash tables configured (established 65536 bind 65536) [ 1.112865] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.116408] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.119704] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.123400] NET: Registered protocol family 1 [ 1.126259] RPC: Registered named UNIX socket transport module. [ 1.128352] RPC: Registered udp transport module. [ 1.130263] RPC: Registered tcp transport module. [ 1.131932] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.134257] NET: Registered protocol family 44 [ 1.135906] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.137903] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.139861] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.142170] PCI: CLS 0 bytes, default 64 [ 1.144697] Unpacking initramfs... [ 2.637574] debug: unmapping init [mem 0xffff8e80fcc54000-0xffff8e80fffbffff] [ 2.642248] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.644925] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.648451] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.174548] Initialise system trusted keyrings [ 3.176432] Key type blacklist registered [ 3.178545] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.189186] zbud: loaded [ 3.193058] *** VALIDATE nfs *** [ 3.194747] *** VALIDATE nfs4 *** [ 3.196754] pstore: using deflate compression [ 3.201118] Platform Keyring initialized [ 3.316380] NET: Registered protocol family 38 [ 3.319245] Key type asymmetric registered [ 3.321303] Asymmetric key parser 'x509' registered [ 3.323377] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.327402] io scheduler mq-deadline registered [ 3.329658] io scheduler kyber registered [ 3.332330] io scheduler bfq registered [ 3.334848] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.338355] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.341758] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.344643] ACPI: Power Button [PWRF] [ 3.349446] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.356925] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.371334] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.377842] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.395404] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.423795] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.456337] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.463273] Non-volatile memory driver v1.3 [ 3.465087] Linux agpgart interface v0.103 [ 3.526446] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 3.532019] vda: detected capacity change from 0 to 74149888 [ 3.559924] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.563951] vdb: detected capacity change from 0 to 1073741824 [ 3.586583] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.589854] vdc: detected capacity change from 0 to 2621440000 [ 3.610755] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.614483] vdd: detected capacity change from 0 to 2621440000 [ 3.635667] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.638594] vde: detected capacity change from 0 to 4294967296 [ 3.660657] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.672934] vdf: detected capacity change from 0 to 4294967296 [ 3.688849] libphy: Fixed MDIO Bus: probed [ 3.697440] usbcore: registered new interface driver usbserial_generic [ 3.700934] usbserial: USB Serial support registered for generic [ 3.703945] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.709922] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.712271] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.715865] mousedev: PS/2 mouse device common for all mice [ 3.720404] rtc_cmos 00:05: RTC can wake from S4 [ 3.724403] rtc_cmos 00:05: registered as rtc0 [ 3.726419] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.730589] intel_pstate: CPU model not supported [ 3.733978] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.736376] hid: raw HID events driver (C) Jiri Kosina [ 3.743657] usbcore: registered new interface driver usbhid [ 3.746533] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.749514] usbhid: USB HID core driver [ 3.749754] drop_monitor: Initializing network drop monitor service [ 3.749906] Initializing XFRM netlink socket [ 3.750331] NET: Registered protocol family 10 [ 3.752036] Segment Routing with IPv6 [ 3.752094] NET: Registered protocol family 17 [ 3.756899] mpls_gso: MPLS GSO support [ 3.759170] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.772983] RAS: Correctable Errors collector initialized. [ 3.775146] AVX version of gcm_enc/dec engaged. [ 3.777139] AES CTR mode by8 optimization enabled [ 3.869663] sched_clock: Marking stable (3869638104, 0)->(4852615394, -982977290) [ 3.874462] registered taskstats version 1 [ 3.879469] Loading compiled-in X.509 certificates [ 3.882102] zswap: loaded using pool lzo/zbud [ 3.909745] Key type big_key registered [ 3.924718] Key type encrypted registered [ 3.926450] ima: No TPM chip found, activating TPM-bypass! [ 3.928920] ima: Allocated hash algorithm: sha1 [ 3.930798] ima: No architecture policies found [ 3.932782] evm: Initialising EVM extended attributes: [ 3.934762] evm: security.selinux [ 3.936233] evm: security.ima [ 3.937454] evm: security.capability [ 3.938782] evm: HMAC attrs: 0x1 [ 3.941321] rtc_cmos 00:05: setting system clock to 2026-06-27 22:56:34 UTC (1782600994) [ 3.947817] debug: unmapping init [mem 0xffffffff8c203000-0xffffffff8c3fffff] [ 3.951766] debug: unmapping init [mem 0xffffffff8af82000-0xffffffff8b258fff] [ 3.960104] Write protecting the kernel read-only data: 28672k [ 3.964114] debug: unmapping init [mem 0xffffffff89603000-0xffffffff897fffff] [ 3.967385] debug: unmapping init [mem 0xffffffff89f14000-0xffffffff89ffffff] [ 4.015843] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.028203] systemd[1]: Detected virtualization kvm. [ 4.030834] systemd[1]: Detected architecture x86-64. [ 4.032960] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.068367] systemd[1]: No hostname configured. [ 4.071319] systemd[1]: Set hostname to . [ 4.073990] random: systemd: uninitialized urandom read (16 bytes read) [ 4.077632] systemd[1]: Initializing machine ID from random generator. [ 4.171067] random: ln: uninitialized urandom read (6 bytes read) [ 4.289924] random: systemd: uninitialized urandom read (16 bytes read) [ 4.294973] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.301879] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.312133] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.127267] device-mapper: uevent: version 1.0.3 [ 5.130340] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.084825] virtio_net virtio0 ens2: renamed from eth0 [ 6.144194] random: fast init done [ 6.278342] scsi host0: ata_piix [ 6.294255] scsi host1: ata_piix [ 6.296215] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.298838] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.959252] random: crng init done [ 10.962089] random: 7 urandom warning(s) missed due to ratelimiting [ 15.071199] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.588820] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.350562] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.385330] SELinux: Disabled at runtime. [ 23.571629] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.611482] systemd[1]: Detected virtualization kvm. [ 23.616274] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.269670] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.278760] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.297587] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.309776] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.328579] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.367895] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.478190] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Disp[ 25.902499] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS atch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 28.174084] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.446185] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.757437] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.697798] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.888794] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 38.949721] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit)[ 40.400697] NFS: Registering the id_resolver key type [ 40.408899] Key type id_resolver registered [ 40.418656] Key type id_legacy registered [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg459-server login: [ 103.098337] hrtimer: interrupt took 8353342 ns [ 107.170140] libcfs: loading out-of-tree module taints kernel. [ 107.237783] Key type ._llcrypt registered [ 107.239278] Key type .llcrypt registered [ 107.398167] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 126.197573] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 128.170197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.197731] alg: No test for adler32 (adler32-zlib) [ 129.991534] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 131.158380] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 133.015325] Key type lgssc registered [ 135.046395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 155.854467] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 203.355275] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 217.476813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 217.547235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.985853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 219.020482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.102350] Lustre: lustre-MDT0000: new disk, initializing [ 219.196206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.234451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.480788] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.225361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.380760] Lustre: 6506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.429515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 238.434441] Lustre: Skipped 1 previous similar message [ 238.598535] Lustre: lustre-MDT0001: new disk, initializing [ 238.687459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.730850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 238.747706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 243.433831] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.487451] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.957731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.252303] Lustre: lustre-OST0000: new disk, initializing [ 258.263068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.277882] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.329307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.254940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.268992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.369222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.653262] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.150260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.367859] Lustre: lustre-OST0001: new disk, initializing [ 280.376874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.386083] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.457879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 286.998274] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.849282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 289.863867] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 290.002894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 299.420067] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.830119] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 310.802698] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing check_logdir /tmp/testlogs/ [ 316.210406] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing yml_node [ 320.695495] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 323.115937] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 325.393484] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 327.093489] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jun 27 19:01:56 EDT 2026 [ 344.286289] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 345.649950] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 361.952621] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.957115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 361.968060] Lustre: Skipped 2 previous similar messages [ 361.985450] Lustre: Skipped 2 previous similar messages [ 366.560827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 366.575238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.606674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.938730] Lustre: server umount lustre-MDT0000 complete [ 371.862420] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782601362 with bad export cookie 2707984685648549291 [ 371.864698] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.873147] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 372.192928] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.200205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.223734] Lustre: Skipped 1 previous similar message [ 372.242344] Lustre: Skipped 5 previous similar messages [ 376.229708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.238591] Lustre: Skipped 1 previous similar message [ 378.471807] Lustre: server umount lustre-MDT0001 complete [ 383.729571] Lustre: server umount lustre-OST0000 complete [ 388.480506] Lustre: server umount lustre-OST0001 complete [ 395.975660] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 406.234788] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 445.002368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.318740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.413942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.569335] Lustre: lustre-MDT0000: new disk, initializing [ 445.710736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.734184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.861346] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 463.589806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.816817] Lustre: 15085:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 463.891592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 463.899401] Lustre: Skipped 1 previous similar message [ 464.028878] Lustre: lustre-MDT0001: new disk, initializing [ 464.162267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 464.199673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 464.216192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 469.984166] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 479.972637] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 486.898262] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.755854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.105260] Lustre: lustre-OST0000: new disk, initializing [ 494.112911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 494.130338] Lustre: 17054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 494.294687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.273324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 495.282679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 495.482537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 500.952430] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 510.793729] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.439479] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 515.684566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 520.159553] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 520.370605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 526.304426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.308094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.324898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 526.338599] Lustre: Skipped 1 previous similar message [ 526.357179] Lustre: Skipped 1 previous similar message [ 528.494821] Lustre: server umount lustre-OST0000 complete [ 536.035633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.046610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.060449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.295447] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.713882] Lustre: server umount lustre-MDT0000 complete [ 550.880442] LustreError: 15932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.927934] LustreError: 15932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 553.173268] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782601543 with bad export cookie 2707984685648550824 [ 553.176158] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.184802] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.605902] Lustre: server umount lustre-MDT0001 complete [ 557.795473] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:05:46 (1782601546) [ 565.548817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.339840] LustreError: 19322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.469664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.538288] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 571.874242] LustreError: 19323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.974713] LustreError: 19322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.487784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.840061] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 592.377534] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 598.544548] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 605.029470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 611.067573] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 617.748501] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 635.362769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 635.368861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.390135] Lustre: Skipped 2 previous similar messages [ 635.391369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 635.402914] Lustre: Skipped 6 previous similar messages [ 641.432067] Lustre: server umount lustre-OST0000 complete [ 646.627596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 652.356626] Lustre: server umount lustre-MDT0000 complete [ 655.842922] LustreError: 19323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.877145] LustreError: 19323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 656.550844] LustreError: 19302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782601647 with bad export cookie 2707984685648551706 [ 656.555775] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.566744] LustreError: 19302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 656.900029] Lustre: server umount lustre-MDT0001 complete [ 669.117266] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 672.256586] Key type lgssc unregistered [ 672.500838] LNet: 22878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.508442] LNetError: 22878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.526376] LNet: Removed LNI 192.168.204.159@tcp [ 673.583172] Key type .llcrypt unregistered [ 673.585762] Key type ._llcrypt unregistered [ 698.047441] Key type ._llcrypt registered [ 698.049636] Key type .llcrypt registered [ 698.140679] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 699.158875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.297371] alg: No test for adler32 (adler32-zlib) [ 700.373075] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 700.609768] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 702.392703] Key type lgssc registered [ 703.460829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.834817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.854792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.327391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.191356] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 721.891874] LustreError: 24294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.915627] LustreError: 24294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 724.081485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 724.432340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 728.291476] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 736.129828] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 742.307361] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 749.079566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.551970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.624933] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 756.733175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 762.596384] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 779.232679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.235937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.251312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 779.277325] Lustre: Skipped 1 previous similar message [ 784.357303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 784.362747] Lustre: Skipped 2 previous similar messages [ 784.725461] Lustre: server umount lustre-OST0000 complete [ 789.485301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.513607] Lustre: Skipped 1 previous similar message [ 789.519803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.529172] Lustre: Skipped 1 previous similar message [ 794.592402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.718534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 799.728127] Lustre: Skipped 2 previous similar messages [ 803.295508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 803.502619] Lustre: server umount lustre-MDT0000 complete [ 804.836596] LustreError: 24293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.865331] LustreError: 24293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 807.885598] LustreError: 24276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782601798 with bad export cookie 6678510489297164684 [ 807.892995] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.899426] LustreError: 24276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 808.300577] Lustre: server umount lustre-MDT0001 complete [ 819.765367] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 822.906990] Key type lgssc unregistered [ 823.227581] LNet: 28270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.243359] LNetError: 28270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.264148] LNet: Removed LNI 192.168.204.159@tcp [ 824.159061] Key type .llcrypt unregistered [ 824.162481] Key type ._llcrypt unregistered [ 845.798864] Key type ._llcrypt registered [ 845.802415] Key type .llcrypt registered [ 845.902950] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 846.967826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.086371] alg: No test for adler32 (adler32-zlib) [ 848.187744] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 848.418987] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 850.135470] Key type lgssc registered [ 851.159662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 858.703331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 858.719990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.140419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.458453] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 870.371771] LustreError: 29682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.399176] LustreError: 29682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 873.299107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.804063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 878.315598] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 886.190209] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 892.618849] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.215225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.480495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.223505] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 918.523855] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 930.038614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 939.488806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 939.504653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.546159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 940.513807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 940.522165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.522437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 945.637827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 945.645107] Lustre: Skipped 1 previous similar message [ 950.240349] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 950.585515] Lustre: server umount lustre-OST0000 complete [ 955.876707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 955.888516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.929756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 955.931822] Lustre: Skipped 1 previous similar message [ 960.825688] Lustre: server umount lustre-MDT0000 complete [ 960.994246] LustreError: 30409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.017245] LustreError: 30409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 964.387086] LustreError: 29663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782601954 with bad export cookie 14088296336242239091 [ 964.407245] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.696025] Lustre: server umount lustre-MDT0001 complete [ 974.131295] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 976.989853] Key type lgssc unregistered [ 977.268695] LNet: 33582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.280963] LNetError: 33582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.300876] LNet: Removed LNI 192.168.204.159@tcp [ 978.203297] Key type .llcrypt unregistered [ 978.207686] Key type ._llcrypt unregistered [ 1004.393260] Key type ._llcrypt registered [ 1004.395854] Key type .llcrypt registered [ 1004.488951] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1005.745379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.773336] alg: No test for adler32 (adler32-zlib) [ 1006.881795] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1007.125555] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1008.783389] Key type lgssc registered [ 1010.005488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1017.311573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1017.327472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.871391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.517425] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1029.095503] LustreError: 34997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.122101] LustreError: 34997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1031.906628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.381985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.077550] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1044.789353] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1050.331825] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1058.190386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.662750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.814915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1066.080175] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1075.489369] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1084.384294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1084.403243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.442960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.456808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1087.476951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.477094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.500711] Lustre: Skipped 1 previous similar message [ 1090.466611] Lustre: server umount lustre-OST0000 complete [ 1097.702576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.707197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.726919] Lustre: Skipped 1 previous similar message [ 1097.742138] Lustre: Skipped 1 previous similar message [ 1101.736259] Lustre: server umount lustre-MDT0000 complete [ 1102.822753] LustreError: 34998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.853383] LustreError: 34998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1105.995928] LustreError: 36841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782602096 with bad export cookie 3286025709761822275 [ 1106.004630] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1106.015070] LustreError: 36841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1106.519379] Lustre: server umount lustre-MDT0001 complete [ 1118.753713] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1121.924451] Key type lgssc unregistered [ 1122.248623] LNet: 38533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.257928] LNetError: 38533:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.274932] LNet: Removed LNI 192.168.204.159@tcp [ 1123.486291] Key type .llcrypt unregistered [ 1123.489684] Key type ._llcrypt unregistered [ 1149.298406] Key type ._llcrypt registered [ 1149.300100] Key type .llcrypt registered [ 1149.440086] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1151.132693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.161325] alg: No test for adler32 (adler32-zlib) [ 1152.240396] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1152.469574] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1154.143560] Key type lgssc registered [ 1155.104280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.748910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.764109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.145222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.662559] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1173.483825] LustreError: 39949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.523349] LustreError: 39949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1177.305880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.629529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.323383] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1190.492602] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1196.161309] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1203.621261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.077336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.211816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1211.461683] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1219.723281] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1229.795982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1229.810093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.834575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.831531] Lustre: lustre-OST0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 1232.872449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.885168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.069594] Lustre: server umount lustre-OST0000 complete [ 1258.466437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.483126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.486035] Lustre: Skipped 2 previous similar messages [ 1261.838756] Lustre: server umount lustre-MDT0000 complete [ 1263.584274] LustreError: 39948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.604884] LustreError: 39948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1265.683186] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782602256 with bad export cookie 9598392398842125433 [ 1265.690818] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.696466] LustreError: 39929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1272.237873] Lustre: server umount lustre-MDT0001 complete [ 1286.153672] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1289.095763] Key type lgssc unregistered [ 1289.360726] LNet: 43656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.392866] LNetError: 43656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.414157] LNet: Removed LNI 192.168.204.159@tcp [ 1290.312553] Key type .llcrypt unregistered [ 1290.318964] Key type ._llcrypt unregistered [ 1311.063877] Key type ._llcrypt registered [ 1311.066049] Key type .llcrypt registered [ 1311.177678] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1312.337250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.349966] alg: No test for adler32 (adler32-zlib) [ 1313.334171] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1313.551263] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1315.231242] Key type lgssc registered [ 1316.331284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1323.905580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1323.939204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.397447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.832209] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1334.779691] LustreError: 45047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.794092] LustreError: 45047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1339.756028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.879910] LustreError: 45048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.182791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1345.566566] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1353.831512] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.148932] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1367.363242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.767497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.855396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1374.256661] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1381.902391] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1391.585290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1391.597230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.606532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.120321] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 1395.684192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.686000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1395.695739] Lustre: Skipped 1 previous similar message [ 1395.706298] Lustre: Skipped 2 previous similar messages [ 1397.457239] Lustre: server umount lustre-MDT0000 complete [ 1399.205629] LustreError: 45043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1401.669628] LustreError: 45028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782602392 with bad export cookie 5436626734852692070 [ 1401.696487] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.328884] LustreError: 45044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.361806] LustreError: 45044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1404.375251] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [ 1405.924128] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.305593] Lustre: server umount lustre-MDT0001 complete [ 1423.495023] Lustre: server umount lustre-OST0000 complete [ 1438.893880] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1441.578993] Key type lgssc unregistered [ 1441.825815] LNet: 48907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.833209] LNetError: 48907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.863657] LNet: Removed LNI 192.168.204.159@tcp [ 1442.649577] Key type .llcrypt unregistered [ 1442.654203] Key type ._llcrypt unregistered [ 1463.196074] Key type ._llcrypt registered [ 1463.198643] Key type .llcrypt registered [ 1463.297594] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1464.121335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.142205] alg: No test for adler32 (adler32-zlib) [ 1465.163747] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1465.353441] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1466.991242] Key type lgssc registered [ 1467.930648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1474.072570] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1474.103044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.631958] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1494.753708] LustreError: 50274:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 1521.503268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.421349] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1533.078088] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1697.694734] Lustre: server umount lustre-OST0000 complete [ 1715.124959] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1717.988928] Key type lgssc unregistered [ 1718.374612] LNet: 51951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.382741] LNetError: 51951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.410396] LNet: Removed LNI 192.168.204.159@tcp [ 1719.313861] Key type .llcrypt unregistered [ 1719.322393] Key type ._llcrypt unregistered [ 1746.360107] Key type ._llcrypt registered [ 1746.363730] Key type .llcrypt registered [ 1746.537778] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1747.896657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.929563] alg: No test for adler32 (adler32-zlib) [ 1748.976590] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1749.281235] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1750.960855] Key type lgssc registered [ 1752.403489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1761.547958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1761.583881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.109824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.696898] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1772.520492] LustreError: 53363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.549346] LustreError: 53363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1776.629465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.175464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1782.391370] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1791.281735] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1797.908655] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1804.760311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.240622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.331167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1811.705382] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1820.123555] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1823.718349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.723257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.731565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.786156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.807900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1826.832446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.959988] Lustre: server umount lustre-OST0000 complete [ 1833.440610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1833.451184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.471781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.478381] Lustre: Skipped 1 previous similar message [ 1837.024481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1837.030187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.921396] Lustre: server umount lustre-MDT0000 complete [ 1842.144762] LustreError: 53364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.167293] LustreError: 53364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1842.716782] LustreError: 53345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782602833 with bad export cookie 9439252465094988327 [ 1842.726825] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.737305] LustreError: 53345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1843.149236] Lustre: server umount lustre-MDT0001 complete [ 1854.411288] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1857.619988] Key type lgssc unregistered [ 1857.936716] LNet: 56895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.949716] LNetError: 56895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.966074] LNet: Removed LNI 192.168.204.159@tcp [ 1859.048496] Key type .llcrypt unregistered [ 1859.054981] Key type ._llcrypt unregistered [ 1882.724493] Key type ._llcrypt registered [ 1882.726666] Key type .llcrypt registered [ 1882.851532] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1883.944198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.000202] alg: No test for adler32 (adler32-zlib) [ 1885.009650] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1885.233410] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1886.895765] Key type lgssc registered [ 1887.944183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.973432] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.996406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1910.495438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1915.615909] LustreError: 58282:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 1942.305858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.457657] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1955.982613] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.286762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.748521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.832498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1969.124743] LustreError: 58307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.148741] LustreError: 58307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1969.150023] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1974.241372] LustreError: 58308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.318849] LustreError: 59319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.339318] LustreError: 59319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1978.036524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.540661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.945521] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1991.997388] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1998.403517] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2003.941535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.956374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.985242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.569319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2009.578275] Lustre: Skipped 2 previous similar messages [ 2014.689805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2014.705943] Lustre: Skipped 2 previous similar messages [ 2015.199985] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2015.310881] Lustre: server umount lustre-OST0000 complete [ 2037.206954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.223445] Lustre: Skipped 1 previous similar message [ 2037.229859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.344808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.351771] Lustre: Skipped 2 previous similar messages [ 2043.659776] Lustre: server umount lustre-MDT0000 complete [ 2045.414877] LustreError: 59322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.437945] LustreError: 59322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2047.467953] LustreError: 58289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782603038 with bad export cookie 4319994233031208316 [ 2047.477060] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.489082] LustreError: 58289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2053.949623] Lustre: server umount lustre-MDT0001 complete [ 2064.231323] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2067.062207] Key type lgssc unregistered [ 2067.392933] LNet: 62028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.403244] LNetError: 62028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.433631] LNet: Removed LNI 192.168.204.159@tcp [ 2068.413130] Key type .llcrypt unregistered [ 2068.415413] Key type ._llcrypt unregistered [ 2090.350788] Key type ._llcrypt registered [ 2090.353615] Key type .llcrypt registered [ 2090.454531] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2091.531943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2091.574504] alg: No test for adler32 (adler32-zlib) [ 2092.603751] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2092.833796] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2094.511213] Key type lgssc registered [ 2095.510455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2102.948911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2102.965620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.298161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2109.081286] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2113.569600] LustreError: 63441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.588180] LustreError: 63441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2118.093927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.515919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.837280] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2132.909958] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.994691] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2146.376635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2146.783140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.907969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2153.999250] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2163.335914] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2182.780635] Lustre: server umount lustre-OST0000 complete [ 2189.795256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.798785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.808025] Lustre: Skipped 1 previous similar message [ 2194.915238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.926026] Lustre: Skipped 2 previous similar messages [ 2197.048757] Lustre: server umount lustre-MDT0000 complete [ 2200.036650] LustreError: 64170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.055059] LustreError: 64170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2200.673383] LustreError: 64178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782603191 with bad export cookie 15358528998608256971 [ 2200.678911] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.681364] LustreError: 64178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2211.448895] Lustre: server umount lustre-MDT0001 complete [ 2222.371106] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2224.910544] Key type lgssc unregistered [ 2225.152946] LNet: 67001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.158826] LNetError: 67001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.171852] LNet: Removed LNI 192.168.204.159@tcp [ 2225.989092] Key type .llcrypt unregistered [ 2225.995076] Key type ._llcrypt unregistered [ 2251.391693] Key type ._llcrypt registered [ 2251.395155] Key type .llcrypt registered [ 2251.545865] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2252.769581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.792487] alg: No test for adler32 (adler32-zlib) [ 2253.913405] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2254.194850] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2255.871177] Key type lgssc registered [ 2256.970673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2266.939616] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 19:34:16 (1782603256) [ 2273.411898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2273.430201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.038817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.503902] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2284.520306] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.540674] LustreError: 68708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2287.529295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.928850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2292.099984] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2299.886692] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2306.419169] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2313.211311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2313.622826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.505797] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2319.793954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2326.478552] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2335.404954] Lustre: Failing over lustre-MDT0000 [ 2335.661462] Lustre: server umount lustre-MDT0000 complete [ 2338.280931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.293390] LustreError: 69997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.304411] LustreError: 69997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2343.394434] LustreError: 69997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.416801] LustreError: 69997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2344.693993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.904925] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2345.051677] Lustre: *** cfs_fail_loc=135, val=0*** [ 2345.127392] LustreError: 71793:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2345.138987] LustreError: 71793:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.159@tcp: cfg command failed: rc = -2 [ 2345.143085] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2345.143085] [ 2345.168545] LustreError: MGC192.168.204.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2345.191097] LustreError: 71781:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2345.203460] LustreError: 71781:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2345.218398] LustreError: 71781:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2345.222701] LustreError: 71781:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2345.240821] Lustre: server umount lustre-MDT0000 complete [ 2345.243365] LustreError: 71781:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2348.519125] LustreError: 68702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.532066] LustreError: 68702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2351.541424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.660620] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2351.851725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.907558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2354.102636] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2356.635362] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2357.229490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2357.264408] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2357.303571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2367.474949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.481269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.495164] Lustre: Skipped 3 previous similar messages [ 2367.510434] Lustre: Skipped 1 previous similar message [ 2371.227123] Lustre: server umount lustre-OST0000 complete [ 2377.709093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2377.714856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.720730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.754702] Lustre: Skipped 1 previous similar message [ 2381.405089] Lustre: server umount lustre-MDT0000 complete [ 2382.816887] LustreError: 68703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.835652] LustreError: 68703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2385.050105] LustreError: 68688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782603375 with bad export cookie 6731159986302302863 [ 2385.074297] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2385.076569] LustreError: 68688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2385.688585] Lustre: server umount lustre-MDT0001 complete [ 2396.429227] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2399.493108] Key type lgssc unregistered [ 2399.839617] LNet: 73573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.847338] LNetError: 73573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.868487] LNet: Removed LNI 192.168.204.159@tcp [ 2400.671619] Key type .llcrypt unregistered [ 2400.673492] Key type ._llcrypt unregistered [ 2429.542542] Key type ._llcrypt registered [ 2429.548597] Key type .llcrypt registered [ 2429.715517] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2431.356065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2431.399711] alg: No test for adler32 (adler32-zlib) [ 2432.589564] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2432.852535] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2434.519193] Key type lgssc registered [ 2435.725644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2442.777411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2442.793359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.275259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.146376] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2453.481834] LustreError: 74990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.508658] LustreError: 74990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2458.118402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.471072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.190601] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2470.808992] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2476.255525] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2483.091939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.451827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.600856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2491.176043] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2499.985628] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2509.824170] Lustre: Failing over lustre-MDT0000 [ 2510.305816] Lustre: server umount lustre-MDT0000 complete [ 2512.351407] LustreError: 77502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.386720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.390884] LustreError: 74986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.415900] Lustre: Skipped 1 previous similar message [ 2517.412913] LustreError: 74985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.434833] LustreError: 74985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2522.513263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.536543] LustreError: 77502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.567022] LustreError: 77502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2522.823334] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.172892] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2523.293584] LustreError: 78078:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2523.306703] LustreError: 78078:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.159@tcp: cfg command failed: rc = -9 [ 2523.314370] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2523.314370] [ 2523.327266] LustreError: MGC192.168.204.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2523.342359] LustreError: 78066:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2523.359863] LustreError: 78066:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2523.376030] LustreError: 78066:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2523.379365] LustreError: 78066:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2523.425221] Lustre: server umount lustre-MDT0000 complete [ 2523.430330] LustreError: 78066:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2531.743179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.899689] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2532.108053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.199447] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2532.584919] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2537.423843] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2537.472136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2537.507642] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2537.573858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2547.681876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.702259] Lustre: Skipped 1 previous similar message [ 2547.710669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2547.719015] Lustre: Skipped 1 previous similar message [ 2551.624481] Lustre: server umount lustre-OST0000 complete [ 2557.927379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.930223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2557.934359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2557.950296] Lustre: Skipped 2 previous similar messages [ 2561.703088] Lustre: server umount lustre-MDT0000 complete [ 2563.044382] LustreError: 75718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.087628] LustreError: 75718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2566.183820] LustreError: 74971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782603556 with bad export cookie 13793042761064698888 [ 2566.194776] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.203480] LustreError: 74971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2566.496513] Lustre: server umount lustre-MDT0001 complete [ 2579.707271] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2584.010362] Key type lgssc unregistered [ 2584.433650] LNet: 79861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2584.439804] LNetError: 79861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2584.462217] LNet: Removed LNI 192.168.204.159@tcp [ 2586.067282] Key type .llcrypt unregistered [ 2586.070126] Key type ._llcrypt unregistered [ 2604.255571] Key type ._llcrypt registered [ 2604.257949] Key type .llcrypt registered [ 2604.365228] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2604.813385] Key type .llcrypt unregistered [ 2604.815700] Key type ._llcrypt unregistered [ 2630.727168] Key type ._llcrypt registered [ 2630.729485] Key type .llcrypt registered [ 2630.857747] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2631.975909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.028990] alg: No test for adler32 (adler32-zlib) [ 2633.115941] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2633.370138] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2635.015188] Key type lgssc registered [ 2636.097084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2642.025518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2642.040272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.331919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.595645] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2652.641153] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.659202] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2653.681732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.996300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2657.822960] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2665.371253] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2670.461137] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2677.691880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.075894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2684.636613] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2688.305328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2693.483972] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2703.856386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.857762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2703.862087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2703.868455] Lustre: Skipped 1 previous similar message [ 2708.840652] Lustre: server umount lustre-OST0000 complete [ 2714.082139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.092992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.102747] Lustre: Skipped 2 previous similar messages [ 2719.142796] Lustre: server umount lustre-MDT0000 complete [ 2719.200444] LustreError: 81762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.226107] LustreError: 81762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2722.854812] LustreError: 82496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782603713 with bad export cookie 5250275556863315250 [ 2722.862639] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.870942] LustreError: 82496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2723.536990] Lustre: server umount lustre-MDT0001 complete [ 2736.255251] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2739.153866] Key type lgssc unregistered [ 2739.505785] LNet: 85295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2739.516717] LNetError: 85295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2739.537553] LNet: Removed LNI 192.168.204.159@tcp [ 2740.429683] Key type .llcrypt unregistered [ 2740.434714] Key type ._llcrypt unregistered [ 2763.241708] Key type ._llcrypt registered [ 2763.245181] Key type .llcrypt registered [ 2763.390499] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2764.626500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2764.671672] alg: No test for adler32 (adler32-zlib) [ 2765.842291] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2766.127455] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2767.824125] Key type lgssc registered [ 2769.224875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2777.607634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2777.664114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.227960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.847034] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2789.348143] LustreError: 86710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.375448] LustreError: 86710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2793.157816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.520786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2798.789605] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2807.662884] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2815.095231] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2824.556563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.034471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.228865] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2841.811419] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2848.568197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2858.978482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.988400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2859.017099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2864.100358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2864.109695] Lustre: Skipped 1 previous similar message [ 2864.990550] Lustre: server umount lustre-OST0000 complete [ 2869.216757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.224335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2869.232166] Lustre: Skipped 2 previous similar messages [ 2869.237923] Lustre: Skipped 1 previous similar message [ 2874.339346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2874.348984] Lustre: Skipped 2 previous similar messages [ 2875.487879] Lustre: server umount lustre-MDT0000 complete [ 2879.459178] LustreError: 86705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.491453] LustreError: 86705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2880.905768] LustreError: 87446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782603871 with bad export cookie 7267119749874456495 [ 2880.908537] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.916405] LustreError: 87446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2881.511797] Lustre: server umount lustre-MDT0001 complete [ 2895.716117] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2898.331443] Key type lgssc unregistered [ 2898.617774] LNet: 90267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2898.627858] LNetError: 90267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.689916] LNet: Removed LNI 192.168.204.159@tcp [ 2900.563844] Key type .llcrypt unregistered [ 2900.567599] Key type ._llcrypt unregistered [ 2925.606776] Key type ._llcrypt registered [ 2925.608508] Key type .llcrypt registered [ 2925.752292] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2927.078334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2927.119047] alg: No test for adler32 (adler32-zlib) [ 2928.236133] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2928.561391] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2930.287183] Key type lgssc registered [ 2931.526286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.349795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.365646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.843658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2945.086227] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2951.147173] LustreError: 91690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.176473] LustreError: 91690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2953.723136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.121571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2958.607147] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2966.290393] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2972.072455] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2979.477763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.744616] LustreError: 93512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.892429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2985.992202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2986.859716] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2995.003665] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3006.436577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.437971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3006.453705] Lustre: Skipped 1 previous similar message [ 3006.461598] Lustre: Skipped 1 previous similar message [ 3011.561783] LustreError: 93512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3011.570684] LustreError: 93512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3011.700467] Lustre: server umount lustre-OST0000 complete [ 3016.679428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.688259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.701708] Lustre: Skipped 1 previous similar message [ 3016.716975] Lustre: Skipped 1 previous similar message [ 3021.794804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.805712] Lustre: Skipped 1 previous similar message [ 3026.914381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3026.925369] Lustre: Skipped 1 previous similar message [ 3029.983178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3030.176111] Lustre: server umount lustre-MDT0000 complete [ 3032.034889] LustreError: 91686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.047423] LustreError: 91686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3034.161742] LustreError: 91672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782604024 with bad export cookie 15953612302594384249 [ 3034.178048] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3034.558722] Lustre: server umount lustre-MDT0001 complete [ 3045.551154] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3048.564184] Key type lgssc unregistered [ 3048.853732] LNet: 95227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3048.858222] LNetError: 95227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3048.875088] LNet: Removed LNI 192.168.204.159@tcp [ 3049.704088] Key type .llcrypt unregistered [ 3049.705832] Key type ._llcrypt unregistered [ 3075.843417] Key type ._llcrypt registered [ 3075.849425] Key type .llcrypt registered [ 3075.974701] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3077.420946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.468735] alg: No test for adler32 (adler32-zlib) [ 3078.513822] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3078.827157] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3080.599397] Key type lgssc registered [ 3082.250876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3092.253801] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3092.267919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.808640] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3112.927340] LustreError: 96616:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 3139.617637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.508276] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3152.647520] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3157.524632] Lustre: server umount lustre-OST0000 complete [ 3165.735834] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:49:14 (1782604154) [ 3173.686335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.329258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.683718] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3179.494634] LustreError: 98257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.527460] LustreError: 98257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3183.586416] LustreError: 98256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.717449] LustreError: 98257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.943812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.625787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3195.526111] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3204.439240] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3211.718819] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3218.635434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.126918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.152973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3225.310181] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3232.788074] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3245.032903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3245.039131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.049143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3245.081431] Lustre: Skipped 1 previous similar message [ 3250.150282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3250.159555] Lustre: Skipped 2 previous similar messages [ 3250.739054] Lustre: server umount lustre-OST0000 complete [ 3255.264752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.269154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.280894] Lustre: Skipped 1 previous similar message [ 3255.292710] Lustre: Skipped 1 previous similar message [ 3260.384703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3260.397541] Lustre: Skipped 1 previous similar message [ 3260.612561] Lustre: server umount lustre-MDT0000 complete [ 3264.346241] LustreError: 98236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782604254 with bad export cookie 8817136488232067554 [ 3264.354857] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3264.379665] LustreError: 98236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3264.864191] Lustre: server umount lustre-MDT0001 complete [ 3276.878961] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3280.154206] Key type lgssc unregistered [ 3280.497075] LNet: 101950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3280.509584] LNetError: 101950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3280.541153] LNet: Removed LNI 192.168.204.159@tcp [ 3281.524798] Key type .llcrypt unregistered [ 3281.529146] Key type ._llcrypt unregistered [ 3322.495691] Key type ._llcrypt registered [ 3322.499204] Key type .llcrypt registered [ 3322.665345] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3324.369599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.426877] alg: No test for adler32 (adler32-zlib) [ 3325.543861] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3325.838390] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3327.655405] Key type lgssc registered [ 3329.375936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3338.655274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.724257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.385399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.202734] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3349.486277] LustreError: 104137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.503589] LustreError: 104137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3353.040609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.360593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3357.337812] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3364.107422] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3370.094465] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3376.309461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.605547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.661392] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3384.827740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3387.450801] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3398.112800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3398.117222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3398.128806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.164551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.165578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.196155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3402.726677] Lustre: server umount lustre-OST0000 complete [ 3409.895403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3409.904566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.935688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3409.945850] Lustre: Skipped 1 previous similar message [ 3412.982107] Lustre: server umount lustre-MDT0000 complete [ 3415.520633] LustreError: 104131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.531662] LustreError: 104131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3416.946154] LustreError: 104117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782604407 with bad export cookie 6257567100484903040 [ 3416.947267] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.961931] LustreError: 104117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3417.300105] Lustre: server umount lustre-MDT0001 complete [ 3427.813932] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3430.733560] Key type lgssc unregistered [ 3431.033476] LNet: 107669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3431.042996] LNetError: 107669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3431.072698] LNet: Removed LNI 192.168.204.159@tcp [ 3431.869696] Key type .llcrypt unregistered [ 3431.877086] Key type ._llcrypt unregistered [ 3451.940714] Key type ._llcrypt registered [ 3451.943173] Key type .llcrypt registered [ 3452.039234] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3453.620176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3453.670351] alg: No test for adler32 (adler32-zlib) [ 3454.903861] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3455.470116] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3457.287199] Key type lgssc registered [ 3459.337806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3467.865765] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3467.898736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.487426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3488.607353] LustreError: 108914:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 3515.296680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.893173] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3528.907569] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.389525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.698872] LustreError: MGC192.168.204.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3535.713504] LustreError: 109938:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3535.725221] LustreError: 109938:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3535.736293] LustreError: 109938:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3535.796371] Lustre: server umount lustre-MDT0000 complete [ 3535.799325] LustreError: 109938:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3557.343178] Lustre: 108249:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782604531/real 1782604531] req@000000008044a3f1 x1869196239572224/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782604547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3557.376825] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.386351] LustreError: 108926:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3577.940923] Lustre: server umount lustre-OST0000 complete [ 3588.449732] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 3599.285719] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3643.186424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.485436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3643.521991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3643.598510] Lustre: lustre-MDT0000: new disk, initializing [ 3643.673863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.702991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3648.317710] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3660.130388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.314845] Lustre: 112735:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3660.394122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3660.401368] Lustre: Skipped 1 previous similar message [ 3660.494422] Lustre: lustre-MDT0001: new disk, initializing [ 3660.622208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3660.691227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3660.702909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3665.651868] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3674.608938] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3681.039873] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3688.033632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.402239] Lustre: lustre-OST0000: new disk, initializing [ 3688.406713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3688.419232] Lustre: 114704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3688.536518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.976945] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3695.161549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3695.170030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3695.304233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3704.201772] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3708.487423] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3708.649945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3713.320885] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3713.640682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3716.072956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3716.074101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.082815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.084631] LustreError: Skipped 1 previous similar message [ 3716.107177] Lustre: Skipped 1 previous similar message [ 3721.196057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.199067] Lustre: Skipped 1 previous similar message [ 3721.774896] Lustre: server umount lustre-OST0000 complete [ 3726.309032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.329714] Lustre: Skipped 1 previous similar message [ 3726.333118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3726.333127] Lustre: Skipped 1 previous similar message [ 3731.426362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.446877] Lustre: Skipped 2 previous similar messages [ 3736.545378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.559160] Lustre: Skipped 1 previous similar message [ 3740.127589] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3740.563899] Lustre: server umount lustre-MDT0000 complete [ 3744.393369] LustreError: 112725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782604734 with bad export cookie 4006855388132252632 [ 3744.411742] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.423110] LustreError: 112725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3744.846639] Lustre: server umount lustre-MDT0001 complete [ 3753.309082] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:59:02 (1782604742) [ 3756.174809] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3773.757673] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 3784.897067] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3823.379359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.594731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3823.614991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3823.719057] Lustre: lustre-MDT0000: new disk, initializing [ 3823.818462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.837428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3828.747499] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3839.968548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.165167] Lustre: 119604:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3840.289782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3840.297543] Lustre: Skipped 1 previous similar message [ 3840.389380] Lustre: lustre-MDT0001: new disk, initializing [ 3840.512525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.552661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3840.564230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3845.217790] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3855.512827] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3862.430447] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3868.875535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3869.220092] Lustre: lustre-OST0000: new disk, initializing [ 3869.224492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3869.233161] Lustre: 121570:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3869.354943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.223465] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3876.942242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3876.967321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3877.136388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3887.752264] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3892.776032] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3893.146479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3898.383518] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3898.631181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3902.944640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3902.948834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.973262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.012696] Lustre: Skipped 1 previous similar message [ 3915.236789] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3915.586746] Lustre: server umount lustre-OST0000 complete [ 3922.405339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3922.430165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.455515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.471277] Lustre: Skipped 5 previous similar messages [ 3926.639950] Lustre: server umount lustre-MDT0000 complete [ 3928.546895] LustreError: 119615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.583960] LustreError: 119615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3931.097774] LustreError: 119595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782604921 with bad export cookie 4006855388132253514 [ 3931.100928] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.117336] LustreError: 119595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3931.444039] Lustre: server umount lustre-MDT0001 complete [ 3941.133455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.824611] LustreError: 123801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.958626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.981226] LustreError: 123802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.284633] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3952.102401] LustreError: 123802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.218410] LustreError: 123801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.485849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.042977] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3970.867599] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3977.961944] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3986.008510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3993.932380] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4002.276399] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4008.932948] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4015.745881] Lustre: server umount lustre-OST0000 complete [ 4023.272517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.278776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.288056] Lustre: Skipped 2 previous similar messages [ 4023.293489] Lustre: Skipped 2 previous similar messages [ 4030.536395] Lustre: server umount lustre-MDT0000 complete [ 4033.506187] Lustre: 108252:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782605008/real 1782605008] req@000000008a201684 x1869196239689472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782605024 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4033.507651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.512792] LustreError: 124804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4033.531774] Lustre: 108252:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4034.606589] LustreError: 123782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605025 with bad export cookie 4006855388132254396 [ 4034.629297] LustreError: 123782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4034.629792] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4035.248115] Lustre: server umount lustre-MDT0001 complete [ 4049.178106] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 4052.744266] Key type lgssc unregistered [ 4053.079548] LNet: 127433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4053.088673] LNetError: 127433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4053.101057] LNet: Removed LNI 192.168.204.159@tcp [ 4054.160522] Key type .llcrypt unregistered [ 4054.165383] Key type ._llcrypt unregistered [ 4075.261284] Key type ._llcrypt registered [ 4075.264807] Key type .llcrypt registered [ 4075.423176] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4092.955860] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4094.168704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4094.282180] alg: No test for adler32 (adler32-zlib) [ 4095.415537] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4095.708117] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 4097.425622] Key type lgssc registered [ 4098.802702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4141.388954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4141.406983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.723834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4142.756777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4142.880177] Lustre: lustre-MDT0000: new disk, initializing [ 4142.985426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4143.012217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4148.071930] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4159.749389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.983824] Lustre: 130595:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4160.078600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4160.085327] Lustre: Skipped 1 previous similar message [ 4160.232844] Lustre: lustre-MDT0001: new disk, initializing [ 4160.444520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4160.499120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4160.519429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4165.706395] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4177.647529] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4185.042725] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4193.247361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.757024] Lustre: lustre-OST0000: new disk, initializing [ 4193.769371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4193.780908] Lustre: 132563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4193.950295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.605664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4201.621337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4201.771714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4201.955249] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4212.444127] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4217.488349] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4217.734881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4222.637966] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4222.929538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4227.560258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4227.560690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4227.568550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4227.574093] LustreError: Skipped 1 previous similar message [ 4227.601941] Lustre: Skipped 1 previous similar message [ 4231.496691] Lustre: server umount lustre-OST0000 complete [ 4237.283609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4237.291780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.305577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.310596] Lustre: Skipped 1 previous similar message [ 4242.812674] Lustre: server umount lustre-MDT0000 complete [ 4247.965948] LustreError: 134106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605238 with bad export cookie 6399080039153863254 [ 4247.968982] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.982782] LustreError: 134106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4248.037475] LustreError: 128814:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 4248.528211] Lustre: server umount lustre-MDT0001 complete [ 4258.764600] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:07:27 (1782605247) [ 4266.379539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.060398] LustreError: 134977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4267.076786] LustreError: 134977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4267.170890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.407358] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4272.611499] LustreError: 134978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.723259] LustreError: 134977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.829365] LustreError: 134978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.927286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.489533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4289.306238] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4297.674973] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4303.768599] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4308.833717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.848026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.855598] Lustre: Skipped 2 previous similar messages [ 4308.872023] Lustre: Skipped 2 previous similar messages [ 4314.083533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4314.100514] Lustre: Skipped 1 previous similar message [ 4319.203151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.225021] Lustre: Skipped 1 previous similar message [ 4320.223313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4320.435762] Lustre: server umount lustre-MDT0000 complete [ 4324.321131] LustreError: 134974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.347789] LustreError: 134974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4325.597136] LustreError: 134958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605316 with bad export cookie 6399080039153864136 [ 4325.599405] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.631284] LustreError: 134958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.123676] Lustre: server umount lustre-MDT0001 complete [ 4335.512309] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:08:44 (1782605324) [ 4343.799970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4359.521499] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4364.646706] LustreError: 137398:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 4390.432542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.933793] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4405.830441] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4408.455831] Lustre: server umount lustre-OST0000 complete [ 4418.684406] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:10:07 (1782605407) [ 4426.741106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.287320] LustreError: 138844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.359182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.922323] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4443.168559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4448.242326] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4457.499228] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4464.243418] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4472.363332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.841641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.846796] Lustre: Skipped 1 previous similar message [ 4480.153050] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4487.721960] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4507.619460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.631596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.651692] Lustre: Skipped 2 previous similar messages [ 4507.657066] Lustre: Skipped 2 previous similar messages [ 4511.718813] Lustre: server umount lustre-MDT0000 complete [ 4512.737949] LustreError: 138844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.799859] LustreError: 138844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4516.812354] LustreError: 138825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605507 with bad export cookie 6399080039153864801 [ 4516.826287] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4516.842028] LustreError: 138825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4517.191622] Lustre: server umount lustre-MDT0001 complete [ 4532.153919] Lustre: server umount lustre-OST0000 complete [ 4542.141172] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:12:11 (1782605531) [ 4550.037604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.655577] LustreError: 142519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4550.739293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.446940] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4565.000093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.425351] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4579.163364] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4585.282394] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4592.560891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4596.043267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4600.602927] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4608.699771] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4613.780466] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4614.088253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4618.801654] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4619.045703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4621.792477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4621.812294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.849745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.890053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4627.724683] Lustre: server umount lustre-OST0000 complete [ 4632.032775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4632.040419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.048198] Lustre: Skipped 1 previous similar message [ 4637.774037] Lustre: server umount lustre-MDT0000 complete [ 4641.249433] LustreError: 142519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4641.260995] LustreError: 142519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4642.023630] LustreError: 142499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605632 with bad export cookie 6399080039153865872 [ 4642.037642] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4642.039757] LustreError: 142499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4642.402650] Lustre: server umount lustre-MDT0001 complete [ 4652.711942] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:14:01 (1782605641) [ 4659.882310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.487908] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4680.607639] LustreError: 146681:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 4706.336144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.345629] Lustre: Skipped 2 previous similar messages [ 4713.539683] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4722.628518] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4730.314159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.852094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4735.888989] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4745.899627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.211315] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4746.220016] Lustre: Skipped 6 previous similar messages [ 4751.512569] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4759.453903] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.522681] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4770.980509] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4771.199146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4775.835618] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4776.098879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4782.053312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4782.070546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.086730] Lustre: Skipped 1 previous similar message [ 4784.745930] Lustre: server umount lustre-MDT0000 complete [ 4787.684946] LustreError: 147718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4787.707880] LustreError: 147718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4788.751656] LustreError: 146688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605779 with bad export cookie 6399080039153866894 [ 4788.770285] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4788.773739] LustreError: 146688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4789.413915] Lustre: server umount lustre-MDT0001 complete [ 4802.787268] Lustre: server umount lustre-OST0000 complete [ 4810.651705] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:16:40 (1782605800) [ 4817.174140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4832.800273] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4837.920637] LustreError: 150877:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 4870.657430] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4879.742344] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4888.233265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.042917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4893.967665] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4903.050991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.398247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4903.410903] Lustre: Skipped 4 previous similar messages [ 4908.794530] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4916.740260] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4923.620773] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4930.815468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.069639] Lustre: lustre-OST0001: new disk, initializing [ 4931.075200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4931.089557] Lustre: 153753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4937.299087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4937.318643] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4937.442135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4937.776534] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4949.294909] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4954.416439] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4954.623721] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4959.764909] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4959.951950] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4963.325766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4963.338208] Lustre: Skipped 2 previous similar messages [ 4968.289663] Lustre: server umount lustre-OST0000 complete [ 4973.536109] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4973.551650] LustreError: Skipped 1 previous similar message [ 4978.687410] Lustre: server umount lustre-OST0001 complete [ 4989.439898] Lustre: server umount lustre-MDT0000 complete [ 4993.568971] LustreError: 150885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782605984 with bad export cookie 6399080039153867846 [ 4993.584341] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4993.585560] LustreError: 150885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4994.010702] Lustre: server umount lustre-MDT0001 complete [ 5021.641481] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:20:10 (1782606010) [ 5023.342996] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5025.155568] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5027.662114] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:20:16 (1782606016) [ 5034.934515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.291861] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5035.320773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5035.520180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5035.531146] Lustre: Skipped 6 previous similar messages [ 5040.346565] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5050.331776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.524607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5050.579826] Lustre: 157310:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5056.253173] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5065.234486] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5071.726834] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5079.944551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.230770] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5086.966173] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5089.975981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5096.128148] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5101.305257] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5101.514796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5106.560346] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5106.881018] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5110.242097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5110.250789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.269130] Lustre: Skipped 5 previous similar messages [ 5110.276694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5110.280551] Lustre: Skipped 8 previous similar messages [ 5115.131658] Lustre: server umount lustre-OST0000 complete [ 5141.772846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5151.044576] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5151.408065] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.429506] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5151.435934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.841196] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5165.731119] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5166.029511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5172.513862] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5172.757567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5179.658827] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff89abe075d000.ost_server_uuid 50 [ 5182.214403] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff89abe075d000.ost_server_uuid in FULL state after 0 sec [ 5182.407980] LustreError: 157317:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.59@tcp) returned error from blocking AST (req@000000004a7824b6 x1869196910931968 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b3714e0c/0x58ce19644f0f25a0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.59@tcp remote: 0x6203ca35cc02dc5b expref: 5 pid: 158571 timeout: 5282 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5182.484935] LustreError: lustre-MDT0000: A client on nid 192.168.204.59@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5182.512315] LustreError: 157308:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.59@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b3714e0c/0x58ce19644f0f25a0 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.59@tcp remote: 0x6203ca35cc02dc5b expref: 6 pid: 158571 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5192.677364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5192.695277] LustreError: Skipped 2 previous similar messages [ 5194.411112] Lustre: server umount lustre-OST0000 complete [ 5204.802607] Lustre: server umount lustre-MDT0000 complete [ 5208.042396] LustreError: 157324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.078877] LustreError: 157324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5209.403061] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782606199 with bad export cookie 6399080039153869029 [ 5209.436103] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5216.083242] Lustre: server umount lustre-MDT0001 complete [ 5228.208083] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5231.301935] Key type lgssc unregistered [ 5231.611760] LNet: 163434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5231.618742] LNetError: 163434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5231.656389] LNet: Removed LNI 192.168.204.159@tcp [ 5232.746423] Key type .llcrypt unregistered [ 5232.751806] Key type ._llcrypt unregistered [ 5260.287225] Key type ._llcrypt registered [ 5260.292185] Key type .llcrypt registered [ 5260.414971] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5261.489296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.505077] alg: No test for adler32 (adler32-zlib) [ 5262.605488] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5262.871662] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5264.559317] Key type lgssc registered [ 5265.897956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5273.751081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5273.779942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.277445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.575836] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5284.836333] LustreError: 164848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.861717] LustreError: 164848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5288.657591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.082342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5293.745689] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5301.352103] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5307.070511] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5315.726113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5316.300671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.402648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5323.178445] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5330.993270] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5341.282241] Lustre: Failing over lustre-MDT0000 [ 5341.618204] Lustre: server umount lustre-MDT0000 complete [ 5344.236375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.238158] LustreError: 166402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.255964] Lustre: Skipped 1 previous similar message [ 5344.272271] LustreError: 166402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5349.347054] LustreError: 167342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.373849] LustreError: 167342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5354.161416] LustreError: 164827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.59@tcp arrived at 1782606344 with bad export cookie 136410965521607713 [ 5354.172705] LustreError: 164827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5354.467279] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5354.491605] LustreError: 164843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5359.583511] Lustre: 164157:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782606334/real 1782606334] req@0000000040a94c22 x1869198134373248/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782606350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5359.588736] LustreError: 165575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.617156] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5359.653611] LustreError: 165575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5361.621520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.825581] LustreError: 164156:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9035c03 x1869198134377984/t0(0) o250->MGC192.168.204.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5369.852108] LustreError: 165575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5369.870455] LustreError: 165575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5370.190531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5370.276895] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5375.065947] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5375.479934] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5375.484470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5377.832643] Lustre: lustre-MDT0000: Denying connection for new client 284f1bd7-3c90-4c0c-8e26-5677d2dfe1d8 (at 192.168.204.59@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5390.306607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5390.313918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5390.329856] Lustre: Skipped 1 previous similar message [ 5390.340282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5392.521762] Lustre: server umount lustre-OST0000 complete [ 5396.508251] LustreError: 168581:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5396.516479] Lustre: 167930:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5396.522465] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5396.537167] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5396.549513] LustreError: 167930:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c1bad222 x1869198134394112/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5396.578391] LustreError: 167930:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5396.597117] LustreError: 167930:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5396.613829] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5396.928698] Lustre: server umount lustre-MDT0000 complete [ 5401.031725] LustreError: 164828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782606391 with bad export cookie 136410965521607818 [ 5401.043393] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5401.053982] LustreError: 164828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5401.090298] LustreError: 164156:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5403.473936] Lustre: server umount lustre-MDT0001 complete [ 5414.894073] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5418.042919] Key type lgssc unregistered [ 5418.351607] LNet: 169333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5418.355758] LNetError: 169333:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5419.431639] LNet: Removed LNI 192.168.204.159@tcp [ 5420.263139] Key type .llcrypt unregistered [ 5420.270930] Key type ._llcrypt unregistered [ 5443.844684] Key type ._llcrypt registered [ 5443.848982] Key type .llcrypt registered [ 5443.994379] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5445.184058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5445.215308] alg: No test for adler32 (adler32-zlib) [ 5446.363965] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5446.609833] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5448.279417] Key type lgssc registered [ 5449.546365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5457.854888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5457.872734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5459.315992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.111275] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5468.648971] LustreError: 170745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.667811] LustreError: 170745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5473.764346] LustreError: 170746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.656982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.114928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5480.316165] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5488.283336] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5494.761103] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5502.941747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5503.426196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5504.453841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5510.969480] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5518.653462] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5530.086811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5530.090823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.107880] LustreError: Skipped 1 previous similar message [ 5530.121363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5530.131288] Lustre: Skipped 1 previous similar message [ 5535.207417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.214106] Lustre: Skipped 2 previous similar messages [ 5540.319896] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5540.323604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5540.349532] Lustre: Skipped 1 previous similar message [ 5540.490746] Lustre: server umount lustre-OST0000 complete [ 5545.448706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.476207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5545.492145] Lustre: Skipped 1 previous similar message [ 5550.563877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5550.573839] Lustre: Skipped 1 previous similar message [ 5551.513199] Lustre: server umount lustre-MDT0000 complete [ 5555.591524] LustreError: 170727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782606546 with bad export cookie 15988148044988647605 [ 5555.598961] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.604400] LustreError: 170727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5555.991558] Lustre: server umount lustre-MDT0001 complete [ 5570.149490] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5573.825399] Key type lgssc unregistered [ 5574.195656] LNet: 174282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5574.203251] LNetError: 174282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5574.222843] LNet: Removed LNI 192.168.204.159@tcp [ 5575.379649] Key type .llcrypt unregistered [ 5575.388097] Key type ._llcrypt unregistered [ 5611.076474] Key type ._llcrypt registered [ 5611.081121] Key type .llcrypt registered [ 5611.299203] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5613.349553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5613.392266] alg: No test for adler32 (adler32-zlib) [ 5614.813579] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5615.239768] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5617.039255] Key type lgssc registered [ 5618.446036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5627.007311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5627.048115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.568264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.048699] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5639.144645] LustreError: 175986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.162715] LustreError: 175986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5641.455398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.819396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.421097] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5655.499653] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5661.817576] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5669.139936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5669.547465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.671306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5676.209870] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5684.414439] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5694.432868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5694.448362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.462361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.513848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.517938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.548675] Lustre: Skipped 1 previous similar message [ 5702.632965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.231974] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5707.529029] Lustre: server umount lustre-OST0000 complete [ 5712.864037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.892840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5712.894022] Lustre: Skipped 1 previous similar message [ 5712.922080] Lustre: Skipped 2 previous similar messages [ 5717.988204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.004114] Lustre: Skipped 1 previous similar message [ 5718.903610] Lustre: server umount lustre-MDT0000 complete [ 5723.108988] LustreError: 175980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.136652] LustreError: 175980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5723.745373] LustreError: 175966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782606714 with bad export cookie 10553403441831320076 [ 5723.749428] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5723.776708] LustreError: 175966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5724.452833] Lustre: server umount lustre-MDT0001 complete [ 5736.946419] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5740.182570] Key type lgssc unregistered [ 5740.600759] LNet: 179519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5740.606391] LNetError: 179519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5740.623595] LNet: Removed LNI 192.168.204.159@tcp [ 5741.779716] Key type .llcrypt unregistered [ 5741.783149] Key type ._llcrypt unregistered [ 5767.863240] Key type ._llcrypt registered [ 5767.871430] Key type .llcrypt registered [ 5768.019720] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5769.154101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5769.246840] alg: No test for adler32 (adler32-zlib) [ 5770.417711] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5770.729212] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5772.471267] Key type lgssc registered [ 5774.040802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5787.640923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5787.675508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.187554] Lustre: *** cfs_fail_loc=135, val=0*** [ 5789.348237] LustreError: 181044:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5789.368393] LustreError: 181044:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.159@tcp: cfg command failed: rc = -2 [ 5789.393781] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5789.393781] [ 5789.414151] LustreError: MGC192.168.204.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5789.439663] LustreError: 181006:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5789.457221] LustreError: 181006:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5789.477065] LustreError: 181006:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5789.489484] LustreError: 181006:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5789.593041] Lustre: server umount lustre-MDT0000 complete [ 5789.599506] LustreError: 181006:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5797.919427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.084238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.587274] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5808.617596] LustreError: 181415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5808.645078] LustreError: 181415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5813.734229] LustreError: 181416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.062716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.392556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5821.280821] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5829.379190] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.924673] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5840.548700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5840.562243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5840.568443] Lustre: Skipped 1 previous similar message [ 5844.796831] Lustre: server umount lustre-MDT0000 complete [ 5845.986712] LustreError: 181411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.003588] LustreError: 181411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5848.454680] LustreError: 181396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782606839 with bad export cookie 8025834381041968856 [ 5848.460109] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5848.474050] LustreError: 181396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5848.906975] Lustre: server umount lustre-MDT0001 complete [ 5860.018752] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5863.446340] Key type lgssc unregistered [ 5863.749723] LNet: 183739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5863.759373] LNetError: 183739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5863.774759] LNet: Removed LNI 192.168.204.159@tcp [ 5864.736984] Key type .llcrypt unregistered [ 5864.738814] Key type ._llcrypt unregistered [ 5889.285702] Key type ._llcrypt registered [ 5889.291568] Key type .llcrypt registered [ 5889.445725] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5889.739425] Key type .llcrypt unregistered [ 5889.746329] Key type ._llcrypt unregistered [ 5906.675451] Key type ._llcrypt registered [ 5906.677858] Key type .llcrypt registered [ 5906.802718] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5907.838262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.874069] alg: No test for adler32 (adler32-zlib) [ 5908.949042] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5909.340472] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5911.151186] Key type lgssc registered [ 5912.670637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5920.874302] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5920.913387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.607542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5941.727353] LustreError: 185874:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 5968.479879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.142979] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5983.768506] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5991.320444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5992.068190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5992.155970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5997.538249] LustreError: 185900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5997.568828] LustreError: 185900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5998.396195] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6002.661532] LustreError: 185899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6002.685994] LustreError: 185899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6007.781888] LustreError: 185900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.823898] LustreError: 185900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6008.572810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.945650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6013.361379] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6021.816067] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6027.960868] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6030.976143] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6044.644418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6044.664312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6044.689823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.164349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.198141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.214189] Lustre: Skipped 1 previous similar message [ 6049.333794] Lustre: server umount lustre-OST0000 complete [ 6054.880547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6054.891698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.906951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6060.513591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6060.519139] Lustre: Skipped 3 previous similar messages [ 6065.635922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6065.645702] Lustre: Skipped 1 previous similar message [ 6067.167139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6067.464869] Lustre: server umount lustre-MDT0000 complete [ 6069.745809] LustreError: 186916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6069.773316] LustreError: 186916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6071.578920] LustreError: 185881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782607062 with bad export cookie 811275761044544371 [ 6071.582362] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6071.597774] LustreError: 185881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6072.183542] Lustre: server umount lustre-MDT0001 complete [ 6082.729891] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6085.325727] Key type lgssc unregistered [ 6085.613564] LNet: 189839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6085.620825] LNetError: 189839:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6086.696879] LNet: Removed LNI 192.168.204.159@tcp [ 6087.608224] Key type .llcrypt unregistered [ 6087.610472] Key type ._llcrypt unregistered [ 6111.359038] Key type ._llcrypt registered [ 6111.361661] Key type .llcrypt registered [ 6111.476926] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6112.326790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6112.391370] alg: No test for adler32 (adler32-zlib) [ 6113.599845] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6113.896520] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6115.663232] Key type lgssc registered [ 6117.034921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.944379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.956646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.442268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.661084] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6135.784084] LustreError: 191253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.802308] LustreError: 191253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6139.223896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.504319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.843510] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6151.081722] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6157.071435] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6164.049630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.397739] LustreError: 193074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.591357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.619462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6170.995521] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6178.968073] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6183.641459] Lustre: Failing over lustre-MDT0000 [ 6183.928295] Lustre: server umount lustre-MDT0000 complete [ 6185.440905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6185.447414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.456757] LustreError: 192262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.965359] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6191.012554] LustreError: 192688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.069855] LustreError: 192688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6196.139373] LustreError: 194000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.349569] Lustre: 190563:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782607176/real 1782607176] req@000000000496af88 x1869199026709504/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782607192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6202.391413] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6204.462979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6206.368798] LustreError: 191249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6206.385283] LustreError: 191249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6211.556836] LustreError: 190562:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000496af88 x1869199026714752/t0(0) o250->MGC192.168.204.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6212.035144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.120882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6216.609301] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6217.201392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6217.229413] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6217.276119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6217.826588] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6223.151269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6229.629852] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6237.668125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.680143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.706139] Lustre: Skipped 2 previous similar messages [ 6237.721736] Lustre: Skipped 1 previous similar message [ 6239.056263] Lustre: server umount lustre-OST0000 complete [ 6247.905356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6247.909202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.916898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6247.932341] Lustre: Skipped 1 previous similar message [ 6249.666580] Lustre: server umount lustre-MDT0000 complete [ 6253.027395] LustreError: 191250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6253.059109] LustreError: 191250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6253.810367] LustreError: 191233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782607244 with bad export cookie 3704884595517361239 [ 6253.811935] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6253.822818] LustreError: 191233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6254.259554] Lustre: server umount lustre-MDT0001 complete [ 6266.335208] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6270.632772] Key type lgssc unregistered [ 6270.967505] LNet: 196097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6270.970815] LNetError: 196097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6270.979708] LNet: Removed LNI 192.168.204.159@tcp [ 6272.061593] Key type .llcrypt unregistered [ 6272.064953] Key type ._llcrypt unregistered [ 6301.913598] Key type ._llcrypt registered [ 6301.923710] Key type .llcrypt registered [ 6302.139294] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6303.634127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6303.708383] alg: No test for adler32 (adler32-zlib) [ 6304.917541] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6305.216065] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6306.927184] Key type lgssc registered [ 6308.169758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6316.990704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6317.040300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.922532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6323.735899] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6329.318667] LustreError: 197510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.338251] LustreError: 197510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6334.130277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.439781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6334.683183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.007536] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6347.837835] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6354.195714] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6360.664316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6361.119293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.215592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6368.492259] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6375.277709] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6378.027194] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6386.687098] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6401.303718] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6410.219088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6410.229423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.249172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6410.271665] Lustre: Skipped 1 previous similar message [ 6410.292266] Lustre: Skipped 1 previous similar message [ 6414.849558] Lustre: server umount lustre-OST0000 complete [ 6420.449694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.467851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.569153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.577458] Lustre: Skipped 2 previous similar messages [ 6429.664585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6429.672068] Lustre: Skipped 1 previous similar message [ 6433.247602] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6433.495581] Lustre: server umount lustre-MDT0000 complete [ 6434.784602] LustreError: 197510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.804495] LustreError: 197510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6437.125505] LustreError: 197492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782607427 with bad export cookie 868234875557792122 [ 6437.136756] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.145573] LustreError: 197492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6437.425248] Lustre: server umount lustre-MDT0001 complete [ 6448.702776] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6451.953362] Key type lgssc unregistered [ 6452.410321] LNet: 201361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6452.435031] LNetError: 201361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6452.470307] LNet: Removed LNI 192.168.204.159@tcp [ 6453.832146] Key type .llcrypt unregistered [ 6453.837757] Key type ._llcrypt unregistered [ 6474.631464] Key type ._llcrypt registered [ 6474.634333] Key type .llcrypt registered [ 6474.732671] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6475.893777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6475.949837] alg: No test for adler32 (adler32-zlib) [ 6477.023215] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6477.273168] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6478.911189] Key type lgssc registered [ 6480.165814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6488.952723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6488.986558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.424481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6494.834829] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6499.810229] LustreError: 202631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6499.834157] LustreError: 202631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6502.674088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.022532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6507.436275] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6514.840054] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6519.809186] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6526.046904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.651345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6531.913165] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6534.834650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6539.856356] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6546.433237] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6556.233061] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6565.888628] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6573.772603] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6584.294985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6584.314320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.333770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.337925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6586.345633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.355735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.093890] Lustre: server umount lustre-OST0000 complete [ 6595.042796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6595.054367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.067147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6595.070229] Lustre: Skipped 1 previous similar message [ 6600.858161] Lustre: server umount lustre-MDT0000 complete [ 6601.698545] LustreError: 204888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.718069] LustreError: 204888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6604.541433] LustreError: 206008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782607595 with bad export cookie 2568813851905228437 [ 6604.542845] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.558845] LustreError: 206008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6605.002304] Lustre: server umount lustre-MDT0001 complete [ 6617.159519] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6620.827224] Key type lgssc unregistered [ 6621.246701] LNet: 207556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6621.254768] LNetError: 207556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6621.286421] LNet: Removed LNI 192.168.204.159@tcp [ 6622.565657] Key type .llcrypt unregistered [ 6622.568862] Key type ._llcrypt unregistered [ 6649.734957] Key type ._llcrypt registered [ 6649.738481] Key type .llcrypt registered [ 6649.862963] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6650.759417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6650.871769] alg: No test for adler32 (adler32-zlib) [ 6651.971970] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6652.262801] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6653.935883] Key type lgssc registered [ 6655.190628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6663.207152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6663.222560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.719405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.157719] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6674.934084] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.948607] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6678.308106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.748823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6683.515947] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6690.953242] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6697.719329] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6704.110461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6704.589192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.757809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6710.961720] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6717.888318] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6734.304773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6734.316359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.328208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6734.344937] Lustre: Skipped 1 previous similar message [ 6734.376257] Lustre: Skipped 1 previous similar message [ 6739.443730] Lustre: server umount lustre-OST0000 complete [ 6744.551290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.554562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6744.577625] Lustre: Skipped 1 previous similar message [ 6749.532404] Lustre: server umount lustre-MDT0000 complete [ 6749.665620] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.701315] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6753.880790] LustreError: 208952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782607744 with bad export cookie 13771890290113113380 [ 6753.883439] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.904442] LustreError: 208952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6754.292078] Lustre: server umount lustre-MDT0001 complete [ 6765.925167] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6769.065750] Key type lgssc unregistered [ 6769.546542] LNet: 212655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6769.552370] LNetError: 212655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6770.602414] LNet: Removed LNI 192.168.204.159@tcp [ 6771.709069] Key type .llcrypt unregistered [ 6771.715835] Key type ._llcrypt unregistered [ 6789.719103] Key type ._llcrypt registered [ 6789.721298] Key type .llcrypt registered [ 6789.815443] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6790.623608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6790.740033] alg: No test for adler32 (adler32-zlib) [ 6791.779030] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6791.975926] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6793.663229] Key type lgssc registered [ 6794.580703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6801.793333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6801.804027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.215064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.328680] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6812.646101] LustreError: 213919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.663417] LustreError: 213919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6815.234780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.686780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6819.908384] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6828.124328] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6833.512196] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6840.044033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6840.484259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6846.777991] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6847.668686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6854.068788] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6868.448787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6868.452673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.462144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6868.484925] Lustre: Skipped 1 previous similar message [ 6873.574665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6873.586259] Lustre: Skipped 1 previous similar message [ 6874.653657] Lustre: server umount lustre-OST0000 complete [ 6878.689885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.718190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.727353] Lustre: Skipped 1 previous similar message [ 6883.809215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6883.817172] Lustre: Skipped 1 previous similar message [ 6884.842081] Lustre: server umount lustre-MDT0000 complete [ 6888.928520] LustreError: 213920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.952661] LustreError: 213920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6889.329170] LustreError: 213899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782607879 with bad export cookie 15309282250326824619 [ 6889.330194] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.354928] LustreError: 213899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6889.715617] Lustre: server umount lustre-MDT0001 complete [ 6900.858328] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6903.970944] Key type lgssc unregistered [ 6904.278872] LNet: 217601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6904.287486] LNetError: 217601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6904.307183] LNet: Removed LNI 192.168.204.159@tcp [ 6905.277079] Key type .llcrypt unregistered [ 6905.279112] Key type ._llcrypt unregistered [ 6923.679494] Key type ._llcrypt registered [ 6923.681472] Key type .llcrypt registered [ 6923.794535] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6924.806291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6924.889431] alg: No test for adler32 (adler32-zlib) [ 6926.034961] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6926.278242] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6927.983153] Key type lgssc registered [ 6929.188269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6937.319855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6937.348836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.776127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.327404] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6949.344713] LustreError: 218865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6949.381467] LustreError: 218865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6953.627844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.103781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6959.140277] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6966.306329] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6971.239483] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6976.910658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.308926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6982.448483] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6983.476985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6989.671252] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6998.158148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6998.266504] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6998.544880] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7005.532836] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7013.982197] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7015.341678] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7027.247882] Lustre: Permanently deactivating lustre-OST0001 [ 7027.260391] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7034.444352] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7039.367501] Lustre: Permanently reactivating lustre-OST0001 [ 7039.379538] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7039.383327] Lustre: Skipped 2 previous similar messages [ 7040.659308] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.692293] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7040.708176] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7040.751339] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7040.779172] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7052.799452] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.809282] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7052.825364] Lustre: Skipped 2 previous similar messages [ 7052.861127] Lustre: Skipped 1 previous similar message [ 7057.713285] Lustre: server umount lustre-OST0001 complete [ 7057.907463] LustreError: 220687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.923528] LustreError: 220687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7063.012630] LustreError: 220686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.014615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.017908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7063.042156] LustreError: 220686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7063.081289] Lustre: Skipped 1 previous similar message [ 7068.132779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7068.134928] LustreError: 220689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.148953] Lustre: Skipped 2 previous similar messages [ 7073.260031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7073.269058] Lustre: Skipped 1 previous similar message [ 7073.269434] LustreError: 220687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.313860] LustreError: 220687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7076.319401] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.529611] Lustre: server umount lustre-OST0000 complete [ 7081.954883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7081.973984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.010938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.729664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.742541] Lustre: Skipped 5 previous similar messages [ 7094.239953] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7094.470748] Lustre: server umount lustre-MDT0000 complete [ 7097.956130] LustreError: 218845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782608088 with bad export cookie 13180367905982899657 [ 7097.958321] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7097.973401] LustreError: 218845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7098.369541] Lustre: server umount lustre-MDT0001 complete [ 7111.328650] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7114.867856] Key type lgssc unregistered [ 7115.274076] LNet: 223916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7115.279457] LNetError: 223916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.305251] LNet: Removed LNI 192.168.204.159@tcp [ 7116.510289] Key type .llcrypt unregistered [ 7116.511983] Key type ._llcrypt unregistered [ 7165.174684] Key type ._llcrypt registered [ 7165.177591] Key type .llcrypt registered [ 7165.317794] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7166.297904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7166.405390] alg: No test for adler32 (adler32-zlib) [ 7167.710453] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 7168.041735] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7169.767726] Key type lgssc registered [ 7171.164752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7179.549350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7179.580438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.918821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7180.948747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7181.168752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.377331] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7195.803038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.133831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7196.225643] Lustre: 226122:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7196.676027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7201.632225] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7210.578592] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7217.374862] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7227.325135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.699911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7227.963419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7235.258971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7236.497918] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7245.645285] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7249.674878] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7254.420808] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7263.551566] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7271.414410] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7287.094832] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7287.102227] Lustre: Skipped 1 previous similar message [ 7306.492276] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7306.498375] Lustre: Skipped 2 previous similar messages [ 7339.323737] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7339.330765] Lustre: Skipped 4 previous similar messages [ 7409.646652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7409.670910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7409.683601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7409.693433] Lustre: Skipped 1 previous similar message [ 7414.320817] Lustre: server umount lustre-OST0000 complete [ 7419.872681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.899512] Lustre: Skipped 1 previous similar message [ 7419.904905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7419.910164] Lustre: Skipped 1 previous similar message [ 7424.997035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.004746] Lustre: Skipped 1 previous similar message [ 7426.639949] Lustre: server umount lustre-MDT0000 complete [ 7431.649315] LustreError: 227981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782608422 with bad export cookie 2210915598427930767 [ 7431.650770] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7431.671150] LustreError: 227981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7432.425129] Lustre: server umount lustre-MDT0001 complete [ 7445.370540] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7449.211165] Key type lgssc unregistered [ 7449.673030] LNet: 230933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7449.680409] LNetError: 230933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7449.713150] LNet: Removed LNI 192.168.204.159@tcp [ 7450.931870] Key type .llcrypt unregistered [ 7450.934409] Key type ._llcrypt unregistered [ 7480.421386] Key type ._llcrypt registered [ 7480.424990] Key type .llcrypt registered [ 7480.559638] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7482.439252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7482.505116] alg: No test for adler32 (adler32-zlib) [ 7483.595830] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 7483.879611] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7485.568094] Key type lgssc registered [ 7486.890715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7496.319667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7496.374045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.937927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.414845] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7508.452309] LustreError: 232358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.473389] LustreError: 232358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7513.455702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.570506] LustreError: 232359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.919310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7518.959167] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7528.126735] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7536.950632] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7545.890343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7546.332167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.596388] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7564.515634] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7568.949357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7589.353314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.371192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.371812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.379868] LustreError: Skipped 1 previous similar message [ 7589.392587] Lustre: Skipped 1 previous similar message [ 7592.412983] Lustre: server umount lustre-OST0000 complete [ 7599.594487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.604881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7599.625809] Lustre: Skipped 2 previous similar messages [ 7599.645793] Lustre: Skipped 1 previous similar message [ 7603.170798] Lustre: server umount lustre-MDT0000 complete [ 7604.711693] LustreError: 232359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7604.745375] LustreError: 232359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7607.528952] LustreError: 232340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782608598 with bad export cookie 16522415481730790906 [ 7607.535406] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7607.541500] LustreError: 232340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7607.905468] Lustre: server umount lustre-MDT0001 complete [ 7620.426485] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7623.373853] Key type lgssc unregistered [ 7623.740215] LNet: 236051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7623.754377] LNetError: 236051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7623.770551] LNet: Removed LNI 192.168.204.159@tcp [ 7624.731747] Key type .llcrypt unregistered [ 7624.736123] Key type ._llcrypt unregistered [ 7650.656454] Key type ._llcrypt registered [ 7650.659194] Key type .llcrypt registered [ 7650.788196] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7651.650996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7651.722133] alg: No test for adler32 (adler32-zlib) [ 7652.893109] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 7653.204787] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7654.945058] Key type lgssc registered [ 7656.222306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7931.080881] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7935.035751] Key type lgssc unregistered [ 7935.419842] LNet: 238074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7935.429752] LNetError: 238074:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7935.462502] LNet: Removed LNI 192.168.204.159@tcp [ 7936.980258] Key type .llcrypt unregistered [ 7936.989512] Key type ._llcrypt unregistered [ 7969.722364] Key type ._llcrypt registered [ 7969.724633] Key type .llcrypt registered [ 7969.875619] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7992.263358] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7994.042151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7994.334954] alg: No test for adler32 (adler32-zlib) [ 7995.615987] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 7995.844287] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7997.575195] Key type lgssc registered [ 7999.317209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8040.332144] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7710 sec ========== 21:10:28 (1782609028) [ 8043.196467] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:10:31 (1782609031) === [ 8047.729884] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:10:36 (1782609036) === [ 8097.050701] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 8100.556460] Key type lgssc unregistered [ 8100.956459] LNet: 243319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8100.964818] LNetError: 243319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8100.991232] LNet: Removed LNI 192.168.204.159@tcp [ 8101.977588] Key type .llcrypt unregistered [ 8101.983265] Key type ._llcrypt unregistered