[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 482230941 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008680] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013055] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.022850] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028638] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.033136] *** VALIDATE proc *** [ 0.034252] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037083] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045139] debug: unmapping init [mem 0xffffffffb4659000-0xffffffffb4660fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056323] rcu: Hierarchical SRCU implementation. [ 0.058435] smp: Bringing up secondary CPUs ... [ 0.059600] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.063511] smp: Brought up 1 node, 4 CPUs [ 0.065012] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.206030] node 0 deferred pages initialised in 137ms [ 0.210074] devtmpfs: initialized [ 0.211314] x86/mm: Memory block size: 128MB [ 0.214891] gcov: version magic: 0x41383552 [ 0.218274] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.222117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.225293] pinctrl core: initialized pinctrl subsystem [ 0.227193] [ 0.227806] ************************************************************* [ 0.230013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232011] ** ** [ 0.234015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.236013] ** ** [ 0.238015] ** This means that this kernel is built to expose internal ** [ 0.241014] ** IOMMU data structures, which may compromise security on ** [ 0.243011] ** your system. ** [ 0.246014] ** ** [ 0.248011] ** If you see this message and you are not debugging the ** [ 0.250014] ** kernel, report this immediately to your vendor! ** [ 0.253021] ** ** [ 0.255014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258019] ************************************************************* [ 0.260787] NET: Registered protocol family 16 [ 0.262585] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.266073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.270192] cpuidle: using governor menu [ 0.271871] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272597] PCI: Using configuration type 1 for base access [ 0.273123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285065] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.290106] cryptd: max_cpu_qlen set to 1000 [ 0.294322] ACPI: Added _OSI(Module Device) [ 0.296017] ACPI: Added _OSI(Processor Device) [ 0.297024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.299019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.305078] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.313305] ACPI: Interpreter enabled [ 0.314078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.316020] ACPI: Using IOAPIC for interrupt routing [ 0.318145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.321453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.333095] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.336069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.339032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.342117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.348514] acpiphp: Slot [2] registered [ 0.350218] acpiphp: Slot [5] registered [ 0.351215] acpiphp: Slot [6] registered [ 0.353187] acpiphp: Slot [7] registered [ 0.355224] acpiphp: Slot [8] registered [ 0.357189] acpiphp: Slot [9] registered [ 0.358264] acpiphp: Slot [10] registered [ 0.360124] acpiphp: Slot [3] registered [ 0.362114] acpiphp: Slot [4] registered [ 0.364201] acpiphp: Slot [11] registered [ 0.366142] acpiphp: Slot [12] registered [ 0.369175] acpiphp: Slot [13] registered [ 0.370181] acpiphp: Slot [14] registered [ 0.372125] acpiphp: Slot [15] registered [ 0.373160] acpiphp: Slot [16] registered [ 0.375179] acpiphp: Slot [17] registered [ 0.376136] acpiphp: Slot [18] registered [ 0.378151] acpiphp: Slot [19] registered [ 0.379140] acpiphp: Slot [20] registered [ 0.381141] acpiphp: Slot [21] registered [ 0.383161] acpiphp: Slot [22] registered [ 0.384153] acpiphp: Slot [23] registered [ 0.386142] acpiphp: Slot [24] registered [ 0.388153] acpiphp: Slot [25] registered [ 0.389154] acpiphp: Slot [26] registered [ 0.391139] acpiphp: Slot [27] registered [ 0.392138] acpiphp: Slot [28] registered [ 0.394110] acpiphp: Slot [29] registered [ 0.395096] acpiphp: Slot [30] registered [ 0.397123] acpiphp: Slot [31] registered [ 0.398098] PCI host bridge to bus 0000:00 [ 0.400023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.402023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.405032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.407026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.410027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.412026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.414189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.417056] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.420451] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.431027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.437018] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.445023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.447585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.448939] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.452050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.456941] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.462017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.475016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.480013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.486063] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.492017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.514026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528506] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.542018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.573629] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.584018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.591000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.602093] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.612040] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.617013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.626021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.647018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.656604] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.664015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.670017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.682016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.691555] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.699014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.706015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.720027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.732903] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.735414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.738366] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.740528] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.743296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.749117] iommu: Default domain type: Passthrough [ 0.751697] SCSI subsystem initialized [ 0.753333] ACPI: bus type USB registered [ 0.755248] usbcore: registered new interface driver usbfs [ 0.757183] usbcore: registered new interface driver hub [ 0.760140] usbcore: registered new device driver usb [ 0.762298] pps_core: LinuxPPS API ver. 1 registered [ 0.764019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.767122] PTP clock support registered [ 0.772137] EDAC MC: Ver: 3.0.0 [ 0.773158] PCI: Using ACPI for IRQ routing [ 0.775140] NetLabel: Initializing [ 0.777063] NetLabel: domain hash size = 128 [ 0.778012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.781133] NetLabel: unlabeled traffic allowed by default [ 0.783470] vgaarb: loaded [ 0.785455] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.790845] clocksource: Switched to clocksource kvm-clock [ 0.899202] VFS: Disk quotas dquot_6.6.0 [ 0.901771] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.904910] *** VALIDATE ramfs *** [ 0.906090] *** VALIDATE hugetlbfs *** [ 0.907906] pnp: PnP ACPI init [ 0.910652] pnp: PnP ACPI: found 6 devices [ 0.927880] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.931915] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.934121] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.935871] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.937936] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.940052] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.943315] NET: Registered protocol family 2 [ 0.945828] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.951699] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.955599] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.960879] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964473] TCP: Hash tables configured (established 65536 bind 65536) [ 0.967546] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.970929] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.973897] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.977304] NET: Registered protocol family 1 [ 0.980210] RPC: Registered named UNIX socket transport module. [ 0.982883] RPC: Registered udp transport module. [ 0.984675] RPC: Registered tcp transport module. [ 0.986500] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.989887] NET: Registered protocol family 44 [ 0.991495] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.993364] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.995218] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.997778] PCI: CLS 0 bytes, default 64 [ 0.999413] Unpacking initramfs... [ 2.436141] debug: unmapping init [mem 0xffff9d09bcc54000-0xffff9d09bffbffff] [ 2.443608] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.446047] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.449441] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.003226] Initialise system trusted keyrings [ 3.004914] Key type blacklist registered [ 3.008384] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.017978] zbud: loaded [ 3.022557] *** VALIDATE nfs *** [ 3.024026] *** VALIDATE nfs4 *** [ 3.026081] pstore: using deflate compression [ 3.029884] Platform Keyring initialized [ 3.141099] NET: Registered protocol family 38 [ 3.143186] Key type asymmetric registered [ 3.145324] Asymmetric key parser 'x509' registered [ 3.147398] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.150367] io scheduler mq-deadline registered [ 3.152635] io scheduler kyber registered [ 3.154387] io scheduler bfq registered [ 3.156608] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.159357] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.162347] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.165203] ACPI: Power Button [PWRF] [ 3.172249] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.179670] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.193501] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.200770] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.214890] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.242235] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.270252] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.275841] Non-volatile memory driver v1.3 [ 3.277752] Linux agpgart interface v0.103 [ 3.310819] virtio_blk virtio1: [vda] 149448 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.315402] vda: detected capacity change from 0 to 76517376 [ 3.332315] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.335156] vdb: detected capacity change from 0 to 1073741824 [ 3.350098] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.353467] vdc: detected capacity change from 0 to 2621440000 [ 3.371054] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373610] vdd: detected capacity change from 0 to 2621440000 [ 3.394078] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396918] vde: detected capacity change from 0 to 4294967296 [ 3.413794] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.416761] vdf: detected capacity change from 0 to 4294967296 [ 3.425727] libphy: Fixed MDIO Bus: probed [ 3.432866] usbcore: registered new interface driver usbserial_generic [ 3.435864] usbserial: USB Serial support registered for generic [ 3.438493] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.443826] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.445941] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.448699] mousedev: PS/2 mouse device common for all mice [ 3.452648] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.452880] rtc_cmos 00:05: RTC can wake from S4 [ 3.462072] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.462397] rtc_cmos 00:05: registered as rtc0 [ 3.467609] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.469640] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.470965] intel_pstate: CPU model not supported [ 3.480361] hid: raw HID events driver (C) Jiri Kosina [ 3.482659] usbcore: registered new interface driver usbhid [ 3.484908] usbhid: USB HID core driver [ 3.486936] drop_monitor: Initializing network drop monitor service [ 3.489356] Initializing XFRM netlink socket [ 3.491062] NET: Registered protocol family 10 [ 3.494590] Segment Routing with IPv6 [ 3.496172] NET: Registered protocol family 17 [ 3.498627] mpls_gso: MPLS GSO support [ 3.503376] RAS: Correctable Errors collector initialized. [ 3.505474] AVX version of gcm_enc/dec engaged. [ 3.507081] AES CTR mode by8 optimization enabled [ 3.588257] sched_clock: Marking stable (3588210046, 0)->(4518169317, -929959271) [ 3.593441] registered taskstats version 1 [ 3.595440] Loading compiled-in X.509 certificates [ 3.597602] zswap: loaded using pool lzo/zbud [ 3.622092] Key type big_key registered [ 3.635384] Key type encrypted registered [ 3.637089] ima: No TPM chip found, activating TPM-bypass! [ 3.639124] ima: Allocated hash algorithm: sha1 [ 3.640884] ima: No architecture policies found [ 3.642665] evm: Initialising EVM extended attributes: [ 3.644444] evm: security.selinux [ 3.645687] evm: security.ima [ 3.646803] evm: security.capability [ 3.648108] evm: HMAC attrs: 0x1 [ 3.650599] rtc_cmos 00:05: setting system clock to 2026-08-22 07:49:03 UTC (1787384943) [ 3.657680] debug: unmapping init [mem 0xffffffffb5603000-0xffffffffb57fffff] [ 3.660816] debug: unmapping init [mem 0xffffffffb4382000-0xffffffffb4658fff] [ 3.670090] Write protecting the kernel read-only data: 28672k [ 3.673643] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 3.676939] debug: unmapping init [mem 0xffffffffb3314000-0xffffffffb33fffff] [ 3.714080] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.723430] systemd[1]: Detected virtualization kvm. [ 3.725575] systemd[1]: Detected architecture x86-64. [ 3.727637] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.750699] systemd[1]: No hostname configured. [ 3.752295] systemd[1]: Set hostname to . [ 3.754529] random: systemd: uninitialized urandom read (16 bytes read) [ 3.757227] systemd[1]: Initializing machine ID from random generator. [ 3.799188] random: ln: uninitialized urandom read (6 bytes read) [ 3.893953] random: systemd: uninitialized urandom read (16 bytes read) [ 3.896419] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.900762] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.906085] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.503582] device-mapper: uevent: version 1.0.3 [ 4.506046] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.190663] random: fast init done [ 5.233149] virtio_net virtio0 ens2: renamed from eth0 [ 5.277977] scsi host0: ata_piix [ 5.292332] scsi host1: ata_piix [ 5.296875] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.298800] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.936551] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.152809] random: crng init done [ 10.154407] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.483045] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.584122] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.836802] SELinux: Disabled at runtime. [ 11.901877] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.910778] systemd[1]: Detected virtualization kvm. [ 11.913083] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.489866] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.493595] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.499273] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.506495] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.510863] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.520626] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.532739] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ 12.685586] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.980960] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.308127] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.390380] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.502456] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.511736] EDAC sbridge: Ver: 1.1.2 [ 15.336689] Key type dns_resolver registered [ 15.646398] NFS: Registering the id_resolver key type [ 15.648819] Key type id_resolver registered [ 15.650691] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg432-server login: [ 43.948818] libcfs: loading out-of-tree module taints kernel. [ 43.969438] Key type ._llcrypt registered [ 43.971203] Key type .llcrypt registered [ 44.030792] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 53.181371] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 53.958132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.968132] alg: No test for adler32 (adler32-zlib) [ 55.000134] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 55.358652] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 57.167743] Key type lgssc registered [ 59.744096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.380966] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 85.223010] hrtimer: interrupt took 4000227 ns [ 124.833652] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 138.491596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.531939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.770419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.804375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 139.914846] Lustre: lustre-MDT0000: new disk, initializing [ 139.995290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.010891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 144.481055] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.841693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.953957] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.013306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.019241] Lustre: Skipped 1 previous similar message [ 159.135508] Lustre: lustre-MDT0001: new disk, initializing [ 159.217238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.246916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.262183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.143225] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.666677] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.124868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.424798] Lustre: lustre-OST0000: new disk, initializing [ 180.429828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.439569] Lustre: 8425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.521633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.375351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.388406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.481519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.919699] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.743493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.015298] Lustre: lustre-OST0001: new disk, initializing [ 202.023016] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.035764] Lustre: 9498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.178965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.946459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.956908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 208.036785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.060821] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.764900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.218486] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.811268] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.253431] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing yml_node [ 245.230110] Lustre: DEBUG MARKER: Client: 2.17.54.112 [ 247.589132] Lustre: DEBUG MARKER: MDS: 2.17.54.112 [ 250.268977] Lustre: DEBUG MARKER: OSS: 2.17.54.112 [ 252.322985] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:53:09 EDT 2026 [ 269.565667] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 271.435506] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.282131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.290878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.294994] Lustre: Skipped 2 previous similar messages [ 290.304146] Lustre: Skipped 3 previous similar messages [ 294.384888] Lustre: server umount lustre-MDT0000 complete [ 297.914118] LustreError: 9497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385237 with bad export cookie 10821551635558292881 [ 297.919347] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.921805] LustreError: 9497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 300.513526] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.518678] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.540375] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 300.569391] Lustre: Skipped 1 previous similar message [ 300.581324] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.591993] Lustre: Skipped 1 previous similar message [ 302.562604] LustreError: 11654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.563641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.589292] LustreError: 11654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 302.611454] Lustre: Skipped 1 previous similar message [ 307.681145] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.681154] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.685957] Lustre: Skipped 1 previous similar message [ 307.717939] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 312.287362] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.625900] Lustre: server umount lustre-MDT0001 complete [ 316.544163] Lustre: server umount lustre-OST0000 complete [ 320.348523] Lustre: server umount lustre-OST0001 complete [ 327.150801] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 338.110351] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 376.626385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.964647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.031433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.112211] Lustre: lustre-MDT0000: new disk, initializing [ 377.242091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.276995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.659298] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 393.870989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.060567] Lustre: 15051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.200114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 394.205523] Lustre: Skipped 1 previous similar message [ 394.390338] Lustre: lustre-MDT0001: new disk, initializing [ 394.538819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 394.587548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 394.603230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 399.343897] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 408.513272] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 414.879681] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 421.820846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.069565] Lustre: lustre-OST0000: new disk, initializing [ 422.074530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 422.081285] Lustre: 17023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 422.154065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.593536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 423.601460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 423.702439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 427.272336] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 435.901564] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 440.363942] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 440.575670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 444.560490] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 444.802948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 449.509358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.526944] Lustre: Skipped 1 previous similar message [ 449.533569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.534026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 449.547955] Lustre: Skipped 1 previous similar message [ 459.755160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.768054] Lustre: Skipped 3 previous similar messages [ 461.279414] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.478936] Lustre: server umount lustre-OST0000 complete [ 466.402147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.408058] LustreError: Skipped 1 previous similar message [ 466.414119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.435021] Lustre: Skipped 1 previous similar message [ 471.476984] Lustre: server umount lustre-MDT0000 complete [ 474.853795] LustreError: 15044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385414 with bad export cookie 10821551635558294414 [ 474.859232] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.868796] LustreError: 15044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.257449] Lustre: server umount lustre-MDT0001 complete [ 478.751854] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:56:56 (1787385416) [ 485.309467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.906096] LustreError: 19288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.007846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.028991] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 491.489268] LustreError: 19289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.584832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.680852] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 511.839250] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 517.307281] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 523.736496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.285468] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 537.931498] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 548.837522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.843284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.863413] Lustre: Skipped 1 previous similar message [ 548.878171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.883836] Lustre: Skipped 4 previous similar messages [ 550.326758] Lustre: server umount lustre-OST0000 complete [ 568.289645] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 568.459983] Lustre: server umount lustre-MDT0000 complete [ 572.024930] LustreError: 19271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385511 with bad export cookie 10821551635558295296 [ 572.036450] LustreError: 19271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 572.038676] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.364698] Lustre: server umount lustre-MDT0001 complete [ 581.728762] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 584.299115] Key type lgssc unregistered [ 584.590247] LNet: 22820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.601517] LNetError: 22820:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.621033] LNet: Removed LNI 192.168.204.132@tcp [ 585.419275] Key type .llcrypt unregistered [ 585.423691] Key type ._llcrypt unregistered [ 606.853845] Key type ._llcrypt registered [ 606.859494] Key type .llcrypt registered [ 607.003443] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 607.678408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.699328] alg: No test for adler32 (adler32-zlib) [ 608.727561] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 609.061363] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 610.767217] Key type lgssc registered [ 612.161595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.115123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 620.176517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.653828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.575074] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 631.776394] LustreError: 24233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.792615] LustreError: 24233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 635.760060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.179120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 640.820490] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 649.300510] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 655.793597] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 663.980441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.426386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.501188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 671.600745] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 680.319501] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 699.457479] Lustre: server umount lustre-OST0000 complete [ 706.532890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.538286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.552258] Lustre: Skipped 1 previous similar message [ 706.568434] Lustre: Skipped 1 previous similar message [ 711.648736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 711.658792] Lustre: Skipped 1 previous similar message [ 713.813576] Lustre: server umount lustre-MDT0000 complete [ 716.769132] LustreError: 24229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.807753] LustreError: 24229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 717.791167] Lustre: 23545:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385641/real 1787385641] req@00000000df8a42e2 x1874209588410624/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787385657 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 717.844522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.865611] LustreError: 25247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.895898] LustreError: 25247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 719.010108] LustreError: 24215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385658 with bad export cookie 14649795133581749544 [ 719.011602] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.040840] LustreError: 24215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 719.436692] Lustre: server umount lustre-MDT0001 complete [ 731.425225] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 734.421365] Key type lgssc unregistered [ 734.790866] LNet: 28225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.814919] LNetError: 28225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.840323] LNet: Removed LNI 192.168.204.132@tcp [ 736.001882] Key type .llcrypt unregistered [ 736.003588] Key type ._llcrypt unregistered [ 764.622751] Key type ._llcrypt registered [ 764.626100] Key type .llcrypt registered [ 764.840849] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 766.689495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.754942] alg: No test for adler32 (adler32-zlib) [ 767.851499] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 768.105763] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 769.807172] Key type lgssc registered [ 771.023693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 779.523660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 779.570469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.122343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.260862] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 790.497509] LustreError: 29640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.514215] LustreError: 29640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 793.426394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.807488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 798.503365] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 805.719848] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.189730] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 823.911757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.238696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 825.283484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 829.755142] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 836.564719] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 843.750761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.751409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 843.753523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 843.769634] Lustre: Skipped 1 previous similar message [ 845.163198] Lustre: server umount lustre-OST0000 complete [ 848.864916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.881793] Lustre: Skipped 1 previous similar message [ 848.882176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 848.897195] Lustre: Skipped 2 previous similar messages [ 853.988722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 854.002585] Lustre: Skipped 1 previous similar message [ 855.029195] Lustre: server umount lustre-MDT0000 complete [ 858.839311] LustreError: 30374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385798 with bad export cookie 7061434227723803833 [ 858.849295] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.852469] LustreError: 30374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 859.284609] Lustre: server umount lustre-MDT0001 complete [ 871.242662] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 874.528959] Key type lgssc unregistered [ 874.966367] LNet: 33518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.979293] LNetError: 33518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.007240] LNet: Removed LNI 192.168.204.132@tcp [ 876.931370] Key type .llcrypt unregistered [ 876.934435] Key type ._llcrypt unregistered [ 898.658802] Key type ._llcrypt registered [ 898.661217] Key type .llcrypt registered [ 898.794533] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 899.576105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.591548] alg: No test for adler32 (adler32-zlib) [ 900.718833] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 901.049774] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 902.711646] Key type lgssc registered [ 903.900336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 911.631655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 911.643736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.026134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.668425] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 922.596382] LustreError: 34912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 922.606063] LustreError: 34912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 925.850406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.149796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 930.597779] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 938.133962] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 944.499692] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 951.055060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.450564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.618278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 957.787917] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 965.071680] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 976.357744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.366599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.372036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.394053] Lustre: Skipped 1 previous similar message [ 978.188452] Lustre: server umount lustre-OST0000 complete [ 982.497389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 982.507945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.521787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 982.532582] Lustre: Skipped 1 previous similar message [ 986.594820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.595456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.713069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.721951] Lustre: Skipped 2 previous similar messages [ 996.319187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 996.836437] LustreError: 36752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.852703] LustreError: 36752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 998.478326] Lustre: server umount lustre-MDT0000 complete [ 1001.953631] LustreError: 34912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.978494] LustreError: 34912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1002.324375] LustreError: 34894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385942 with bad export cookie 267476990939744967 [ 1002.331982] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.335601] LustreError: 34894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.593170] Lustre: server umount lustre-MDT0001 complete [ 1014.361260] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1017.126827] Key type lgssc unregistered [ 1017.457377] LNet: 38448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.477273] LNetError: 38448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.504377] LNet: Removed LNI 192.168.204.132@tcp [ 1018.399196] Key type .llcrypt unregistered [ 1018.401227] Key type ._llcrypt unregistered [ 1041.556979] Key type ._llcrypt registered [ 1041.560952] Key type .llcrypt registered [ 1041.653329] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1042.823105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1042.843066] alg: No test for adler32 (adler32-zlib) [ 1043.918680] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1044.139888] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1045.831248] Key type lgssc registered [ 1047.167681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.069854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.089349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.640131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.945294] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1066.978583] LustreError: 39862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.009975] LustreError: 39862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1069.514529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.809798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.669922] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1082.674636] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1089.125361] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1096.336300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1096.808953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.937799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1102.959246] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1111.171551] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1122.790479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1122.800882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.818972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1123.374259] Lustre: lustre-OST0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 1124.836718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.851042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.862549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.883692] Lustre: Skipped 1 previous similar message [ 1126.710108] Lustre: server umount lustre-OST0000 complete [ 1146.849706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1146.855412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.871239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.409380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.317426] Lustre: server umount lustre-MDT0000 complete [ 1154.532984] LustreError: 39858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.574988] LustreError: 39858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1156.110656] LustreError: 39843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386095 with bad export cookie 2346607601141914847 [ 1156.119446] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.122402] LustreError: 39843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1162.603326] Lustre: server umount lustre-MDT0001 complete [ 1175.607394] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1178.903491] Key type lgssc unregistered [ 1179.392591] LNet: 43576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.407652] LNetError: 43576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.471023] LNet: Removed LNI 192.168.204.132@tcp [ 1181.569241] Key type .llcrypt unregistered [ 1181.571074] Key type ._llcrypt unregistered [ 1205.430339] Key type ._llcrypt registered [ 1205.435056] Key type .llcrypt registered [ 1205.524742] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1206.345642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.364305] alg: No test for adler32 (adler32-zlib) [ 1207.446992] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1207.672017] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1209.359322] Key type lgssc registered [ 1210.557688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1218.335403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1218.362994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.780868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.351729] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1229.285967] LustreError: 44993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1229.307763] LustreError: 44993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1232.942907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1233.343257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.034957] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1244.046713] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1250.293894] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1258.250242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1258.823283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.962412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1266.509992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1274.221554] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1281.767493] Lustre: server umount lustre-MDT0000 complete [ 1282.082724] LustreError: 44989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.577579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1284.587241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.612328] LustreError: 44992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1285.603946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.622346] Lustre: Skipped 1 previous similar message [ 1286.111041] LustreError: 46836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386225 with bad export cookie 6272524839636077432 [ 1286.122952] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1286.127418] LustreError: 46836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1287.177589] LustreError: 44988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.212773] LustreError: 44988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1287.231723] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 1290.720680] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.737706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1291.745826] LustreError: 44987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.747248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1291.780157] LustreError: 44987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1292.696882] Lustre: server umount lustre-MDT0001 complete [ 1314.516551] Lustre: server umount lustre-OST0000 complete [ 1333.238175] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1336.250734] Key type lgssc unregistered [ 1336.600180] LNet: 48856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.607811] LNetError: 48856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.627624] LNet: Removed LNI 192.168.204.132@tcp [ 1337.563733] Key type .llcrypt unregistered [ 1337.573392] Key type ._llcrypt unregistered [ 1363.572876] Key type ._llcrypt registered [ 1363.576647] Key type .llcrypt registered [ 1363.666611] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1364.616018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.643982] alg: No test for adler32 (adler32-zlib) [ 1365.746832] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1365.965071] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1367.607557] Key type lgssc registered [ 1368.586903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1375.753253] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1375.775420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1391.391438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1396.511355] LustreError: 50244:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 1423.201771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.711441] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1438.566632] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1606.475326] Lustre: server umount lustre-OST0000 complete [ 1623.183427] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1626.102494] Key type lgssc unregistered [ 1626.380721] LNet: 51919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.392725] LNetError: 51919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.413198] LNet: Removed LNI 192.168.204.132@tcp [ 1627.529664] Key type .llcrypt unregistered [ 1627.537878] Key type ._llcrypt unregistered [ 1654.195509] Key type ._llcrypt registered [ 1654.198263] Key type .llcrypt registered [ 1654.275356] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1655.889732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.925134] alg: No test for adler32 (adler32-zlib) [ 1657.119635] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1657.458475] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1659.207644] Key type lgssc registered [ 1660.491975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1668.542534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1668.564208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.089907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.741400] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1679.337202] LustreError: 53336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.381892] LustreError: 53336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1684.449216] LustreError: 53335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.667989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.093624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.570531] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1698.575258] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1704.281709] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1711.606160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.955531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.057379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1718.973529] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1727.161549] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.530138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1730.541191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.556079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.629900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1734.631212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1734.648530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.674948] Lustre: Skipped 1 previous similar message [ 1736.135248] Lustre: server umount lustre-OST0000 complete [ 1741.281835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1741.296306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.315549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.865989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.872155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.885409] Lustre: Skipped 1 previous similar message [ 1746.794498] Lustre: server umount lustre-MDT0000 complete [ 1749.990961] LustreError: 53335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.029192] LustreError: 53335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1750.847426] LustreError: 53314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386690 with bad export cookie 17234822348186205428 [ 1750.855381] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1750.874793] LustreError: 53314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1751.180707] Lustre: server umount lustre-MDT0001 complete [ 1762.932294] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1765.756858] Key type lgssc unregistered [ 1766.116251] LNet: 56866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.124923] LNetError: 56866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.146563] LNet: Removed LNI 192.168.204.132@tcp [ 1767.121608] Key type .llcrypt unregistered [ 1767.129638] Key type ._llcrypt unregistered [ 1794.865224] Key type ._llcrypt registered [ 1794.868709] Key type .llcrypt registered [ 1795.017222] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1796.101593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1796.175740] alg: No test for adler32 (adler32-zlib) [ 1797.545574] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 1797.865132] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1799.583187] Key type lgssc registered [ 1800.849500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.022329] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1810.040655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1825.633155] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1830.751490] LustreError: 58257:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 1857.503295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.427193] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1872.739457] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1880.468695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.309722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.408170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1886.693820] LustreError: 58282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1886.725892] LustreError: 58282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1886.825146] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1891.815373] LustreError: 58283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.836828] LustreError: 59294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.864533] LustreError: 59294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1896.622521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.040622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.604714] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1911.334439] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1917.972113] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.534027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.548633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.555736] LustreError: Skipped 1 previous similar message [ 1922.572563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.580061] Lustre: Skipped 1 previous similar message [ 1922.619318] Lustre: Skipped 1 previous similar message [ 1927.650128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.670036] Lustre: Skipped 1 previous similar message [ 1932.771220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.790531] Lustre: Skipped 1 previous similar message [ 1934.303226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1934.460648] Lustre: server umount lustre-OST0000 complete [ 1962.409548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.423883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.431045] Lustre: Skipped 1 previous similar message [ 1965.737904] Lustre: server umount lustre-MDT0000 complete [ 1967.586830] LustreError: 59299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.614441] LustreError: 59299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1969.917316] LustreError: 58263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386909 with bad export cookie 3835015282873863159 [ 1969.943613] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.428809] Lustre: server umount lustre-MDT0001 complete [ 1987.956762] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1991.401135] Key type lgssc unregistered [ 1991.781703] LNet: 62003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.806065] LNetError: 62003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.840492] LNet: Removed LNI 192.168.204.132@tcp [ 1993.069788] Key type .llcrypt unregistered [ 1993.071817] Key type ._llcrypt unregistered [ 2019.040380] Key type ._llcrypt registered [ 2019.043694] Key type .llcrypt registered [ 2019.167795] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2020.397253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.430211] alg: No test for adler32 (adler32-zlib) [ 2021.474209] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2021.820211] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2023.615513] Key type lgssc registered [ 2024.908048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.069176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.088573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.610241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.139897] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2044.896312] LustreError: 63415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.907133] LustreError: 63415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2048.326708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.885907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.631983] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2061.625201] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.601517] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2074.430701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2074.993289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.113897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2081.998662] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2090.306909] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2108.896353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2108.898894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.909519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2108.917853] Lustre: Skipped 1 previous similar message [ 2111.133325] Lustre: server umount lustre-OST0000 complete [ 2115.552769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2115.556527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.585218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.589320] Lustre: Skipped 1 previous similar message [ 2119.138836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.144205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.161300] Lustre: Skipped 1 previous similar message [ 2121.696755] Lustre: server umount lustre-MDT0000 complete [ 2124.271342] LustreError: 64580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.286435] LustreError: 64580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2126.836676] LustreError: 63397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387066 with bad export cookie 9368210663524440553 [ 2126.839944] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.843181] LustreError: 63397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2127.481326] Lustre: server umount lustre-MDT0001 complete [ 2139.667410] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2142.453057] Key type lgssc unregistered [ 2142.787659] LNet: 66973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.792268] LNetError: 66973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.825865] LNet: Removed LNI 192.168.204.132@tcp [ 2143.691247] Key type .llcrypt unregistered [ 2143.694987] Key type ._llcrypt unregistered [ 2172.276597] Key type ._llcrypt registered [ 2172.282328] Key type .llcrypt registered [ 2172.439785] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2173.972715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2174.034797] alg: No test for adler32 (adler32-zlib) [ 2175.192804] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2175.564752] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2177.287333] Key type lgssc registered [ 2178.744846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2190.605151] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:25:27 (1787387127) [ 2199.014578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.038291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.627800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.601398] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2210.788225] LustreError: 68679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.804106] LustreError: 68679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2215.905383] LustreError: 68680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.267041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.718689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.033276] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2230.675216] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2237.466577] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2244.611032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2245.219931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.308505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2253.221522] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2261.763035] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2269.328058] Lustre: Failing over lustre-MDT0000 [ 2269.752374] LustreError: 68674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.671651] Lustre: server umount lustre-MDT0000 complete [ 2273.252823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2273.268874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.280767] LustreError: 68674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.326591] Lustre: Skipped 2 previous similar messages [ 2273.352946] LustreError: 68674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2278.373199] LustreError: 68679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.406923] LustreError: 68679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2283.402952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.578943] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.773741] Lustre: *** cfs_fail_loc=135, val=0*** [ 2283.971164] LustreError: 71766:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2284.016837] LustreError: 71766:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -2 [ 2284.039887] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2284.039887] [ 2284.078257] LustreError: MGC192.168.204.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2284.117133] LustreError: 71754:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2284.140089] LustreError: 71754:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2284.162336] LustreError: 71754:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2284.191806] LustreError: 71754:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2284.282260] Lustre: server umount lustre-MDT0000 complete [ 2284.293120] LustreError: 71754:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2288.611649] LustreError: 68674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.644708] LustreError: 68674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2293.460922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.568546] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.581210] LustreError: 68671:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2293.735700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2293.749224] Lustre: Skipped 2 previous similar messages [ 2293.953539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.031478] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2295.306960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2299.286444] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2299.376140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2299.451938] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2299.528032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2309.602262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.637061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2314.209496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2314.228877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.253679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.739852] Lustre: server umount lustre-OST0000 complete [ 2319.842557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.844073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.856317] Lustre: Skipped 1 previous similar message [ 2319.870192] Lustre: Skipped 2 previous similar messages [ 2324.960863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2324.969528] Lustre: Skipped 1 previous similar message [ 2334.175163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2334.180089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.194025] Lustre: Skipped 3 previous similar messages [ 2334.445513] Lustre: server umount lustre-MDT0000 complete [ 2338.335105] LustreError: 68659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387278 with bad export cookie 1814028130510494234 [ 2338.337546] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.343494] LustreError: 68659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2338.807721] Lustre: server umount lustre-MDT0001 complete [ 2349.598084] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2352.215758] Key type lgssc unregistered [ 2352.472566] LNet: 73546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.484591] LNetError: 73546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.501290] LNet: Removed LNI 192.168.204.132@tcp [ 2353.327374] Key type .llcrypt unregistered [ 2353.329483] Key type ._llcrypt unregistered [ 2378.594329] Key type ._llcrypt registered [ 2378.597331] Key type .llcrypt registered [ 2378.725995] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2379.779231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.828599] alg: No test for adler32 (adler32-zlib) [ 2380.908876] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2381.158376] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2382.848080] Key type lgssc registered [ 2384.276540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2393.063477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2393.101174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.712879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.745857] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2404.846194] LustreError: 74962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.864668] LustreError: 74962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2409.579956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.062866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2415.070555] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2423.241094] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2429.340515] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2436.610445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2437.059309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.131160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2445.796979] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2456.358522] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2463.169522] Lustre: Failing over lustre-MDT0000 [ 2463.576429] Lustre: server umount lustre-MDT0000 complete [ 2464.282991] LustreError: 76923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.747374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2464.781291] Lustre: Skipped 2 previous similar messages [ 2469.377741] LustreError: 76923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.404608] LustreError: 76923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2474.506335] LustreError: 74959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.522028] LustreError: 74959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2474.950048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.054209] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2475.226507] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2475.368445] LustreError: 78051:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2475.377411] LustreError: 78051:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -9 [ 2475.395679] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2475.395679] [ 2475.408220] LustreError: MGC192.168.204.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2475.420891] LustreError: 78040:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2475.425443] LustreError: 78040:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2475.430616] LustreError: 78040:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2475.437255] LustreError: 78040:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2475.452363] Lustre: server umount lustre-MDT0000 complete [ 2475.455884] LustreError: 78040:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2479.615990] LustreError: 74959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.636943] LustreError: 74959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2482.777137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.983055] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.256359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.312996] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2485.753449] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2488.307827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2488.335337] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2488.381087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2488.570844] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2498.536046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.567547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.574092] Lustre: Skipped 1 previous similar message [ 2503.656669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.670105] Lustre: Skipped 1 previous similar message [ 2504.093901] Lustre: server umount lustre-OST0000 complete [ 2508.768574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2508.776226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.788144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.817617] Lustre: Skipped 2 previous similar messages [ 2513.893349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.909076] Lustre: Skipped 2 previous similar messages [ 2514.853579] Lustre: server umount lustre-MDT0000 complete [ 2519.016273] LustreError: 75692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.045127] LustreError: 75692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2519.576226] LustreError: 74942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387459 with bad export cookie 1055993578699756744 [ 2519.584573] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.596680] LustreError: 74942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2519.929602] Lustre: server umount lustre-MDT0001 complete [ 2533.730582] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2536.954267] Key type lgssc unregistered [ 2537.392811] LNet: 79840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.406856] LNetError: 79840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.426372] LNet: Removed LNI 192.168.204.132@tcp [ 2538.467717] Key type .llcrypt unregistered [ 2538.470807] Key type ._llcrypt unregistered [ 2556.174699] Key type ._llcrypt registered [ 2556.176275] Key type .llcrypt registered [ 2556.294319] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2556.627549] Key type .llcrypt unregistered [ 2556.633562] Key type ._llcrypt unregistered [ 2588.767099] Key type ._llcrypt registered [ 2588.769223] Key type .llcrypt registered [ 2588.827727] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2590.099491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2590.111398] alg: No test for adler32 (adler32-zlib) [ 2591.464915] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2591.890968] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2593.751469] Key type lgssc registered [ 2595.275363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.850316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2604.872787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.370409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.734297] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2616.809144] LustreError: 81746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.835222] LustreError: 81746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2621.541654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.036290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.027793] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2635.060884] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2641.888560] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2651.315096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.121627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.135162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2660.455276] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2669.516878] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2682.852208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.879183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2682.881714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.897573] Lustre: Skipped 1 previous similar message [ 2686.464705] Lustre: server umount lustre-OST0000 complete [ 2693.092540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.095259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.107557] Lustre: Skipped 2 previous similar messages [ 2693.125805] Lustre: Skipped 1 previous similar message [ 2697.149283] Lustre: server umount lustre-MDT0000 complete [ 2698.209390] LustreError: 82729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.222762] LustreError: 82729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2701.029922] LustreError: 81725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387640 with bad export cookie 12917796849155523147 [ 2701.040420] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2701.041501] LustreError: 81725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2701.390985] Lustre: server umount lustre-MDT0001 complete [ 2712.532739] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2715.911638] Key type lgssc unregistered [ 2716.196699] LNet: 85279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2716.212203] LNetError: 85279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2716.246824] LNet: Removed LNI 192.168.204.132@tcp [ 2717.254698] Key type .llcrypt unregistered [ 2717.258280] Key type ._llcrypt unregistered [ 2746.776904] Key type ._llcrypt registered [ 2746.783136] Key type .llcrypt registered [ 2746.948135] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2747.960116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2748.039839] alg: No test for adler32 (adler32-zlib) [ 2749.240790] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2749.664544] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2751.400653] Key type lgssc registered [ 2753.158452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2762.189866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2762.206763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.818990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.664531] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2773.984220] LustreError: 86694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.994594] LustreError: 86694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2779.117629] LustreError: 86695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.004743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.515953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.374549] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2799.366480] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2806.911662] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2815.031990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.381050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.424472] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2832.243555] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2833.931625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2844.131845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.136801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.141384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.149078] Lustre: Skipped 1 previous similar message [ 2849.252269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.256888] Lustre: Skipped 2 previous similar messages [ 2849.597575] Lustre: server umount lustre-OST0000 complete [ 2854.374604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.393994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.404303] Lustre: Skipped 1 previous similar message [ 2859.492592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.502088] Lustre: Skipped 1 previous similar message [ 2860.042780] Lustre: server umount lustre-MDT0000 complete [ 2864.248203] LustreError: 89478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387804 with bad export cookie 7837584392541518811 [ 2864.250305] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2864.266140] LustreError: 89478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2864.741883] Lustre: server umount lustre-MDT0001 complete [ 2877.909516] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2882.211827] Key type lgssc unregistered [ 2882.739247] LNet: 90230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2882.759386] LNetError: 90230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2882.809220] LNet: Removed LNI 192.168.204.132@tcp [ 2883.893624] Key type .llcrypt unregistered [ 2883.901629] Key type ._llcrypt unregistered [ 2915.100552] Key type ._llcrypt registered [ 2915.113280] Key type .llcrypt registered [ 2915.271599] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2917.072615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2917.123042] alg: No test for adler32 (adler32-zlib) [ 2918.331388] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 2918.550617] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2920.215725] Key type lgssc registered [ 2921.936470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.569435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2930.620732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.287852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.708330] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2942.442279] LustreError: 91645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.484825] LustreError: 91645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2947.552096] LustreError: 91646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.292249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.709353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.848933] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2961.532073] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2968.207480] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2975.302377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.911320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.995218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2982.138134] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2990.079293] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3005.941521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3005.957281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.982225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3008.078889] Lustre: server umount lustre-OST0000 complete [ 3014.113160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.119631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.134548] Lustre: Skipped 1 previous similar message [ 3014.146628] Lustre: Skipped 1 previous similar message [ 3019.234298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.244736] Lustre: Skipped 1 previous similar message [ 3024.353211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.374433] Lustre: Skipped 1 previous similar message [ 3024.991246] Lustre: 90958:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387948/real 1787387948] req@000000005a414e78 x1874212010620160/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787387964 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3025.028184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.399521] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3026.708268] Lustre: server umount lustre-MDT0000 complete [ 3029.407325] Lustre: 90956:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387953/real 1787387953] req@00000000d76710fe x1874212010622464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3030.143379] LustreError: 93489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387969 with bad export cookie 10016137841325366103 [ 3030.152044] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.152454] LustreError: 93489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3030.435761] Lustre: server umount lustre-MDT0001 complete [ 3041.201370] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3043.846982] Key type lgssc unregistered [ 3044.154043] LNet: 95198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3044.164562] LNetError: 95198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3044.186795] LNet: Removed LNI 192.168.204.132@tcp [ 3045.241162] Key type .llcrypt unregistered [ 3045.245709] Key type ._llcrypt unregistered [ 3070.749517] Key type ._llcrypt registered [ 3070.752532] Key type .llcrypt registered [ 3070.908428] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3072.098271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3072.178652] alg: No test for adler32 (adler32-zlib) [ 3073.186450] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3073.475644] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3075.119158] Key type lgssc registered [ 3076.327321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3084.523417] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3084.551118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.127698] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3105.248215] LustreError: 96588:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 3131.935238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.083873] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3145.380465] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3150.935729] Lustre: server umount lustre-OST0000 complete [ 3159.309466] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:41:36 (1787388096) [ 3166.329428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.735972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.910745] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3171.816778] LustreError: 98227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.856444] LustreError: 98227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3176.930053] LustreError: 98226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3179.414833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.742770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3184.220988] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3191.772863] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3197.445709] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3204.109697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3204.461107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3210.856273] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3211.700765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3218.244968] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3231.202731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.219511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.233038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.225363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3232.231441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.248161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3235.152231] Lustre: server umount lustre-OST0000 complete [ 3241.440554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3241.455213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.499561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.503350] Lustre: Skipped 1 previous similar message [ 3244.857393] Lustre: server umount lustre-MDT0000 complete [ 3247.585094] LustreError: 99270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.612250] LustreError: 99270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3248.775117] LustreError: 101167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388188 with bad export cookie 13792291713788696128 [ 3248.776530] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3248.784905] LustreError: 101167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3249.208395] Lustre: server umount lustre-MDT0001 complete [ 3261.573536] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3265.144735] Key type lgssc unregistered [ 3265.564830] LNet: 101919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3265.575653] LNetError: 101919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3265.604075] LNet: Removed LNI 192.168.204.132@tcp [ 3266.547469] Key type .llcrypt unregistered [ 3266.550775] Key type ._llcrypt unregistered [ 3308.944468] Key type ._llcrypt registered [ 3308.949933] Key type .llcrypt registered [ 3309.069150] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3310.490924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3310.553257] alg: No test for adler32 (adler32-zlib) [ 3311.551781] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3311.829945] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3313.529906] Key type lgssc registered [ 3314.654769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3322.252604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3322.282114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.790160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.117188] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3334.120508] LustreError: 104104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.142445] LustreError: 104104:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3336.619282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.068169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.493354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3349.005988] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3354.060078] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3360.370992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.769352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.861108] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3368.967476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3374.918927] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3384.296321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.321791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.323484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.202338] Lustre: server umount lustre-OST0000 complete [ 3393.505130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3393.517194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.543763] Lustre: Skipped 1 previous similar message [ 3393.550149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.567241] Lustre: Skipped 1 previous similar message [ 3394.533754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.649069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.658126] Lustre: Skipped 3 previous similar messages [ 3399.991486] Lustre: server umount lustre-MDT0000 complete [ 3404.223702] LustreError: 104086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388344 with bad export cookie 13266030653140523896 [ 3404.225996] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3404.238623] LustreError: 104086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3404.788672] Lustre: server umount lustre-MDT0001 complete [ 3417.922224] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3420.591652] Key type lgssc unregistered [ 3420.850833] LNet: 107638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3420.855354] LNetError: 107638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3420.868050] LNet: Removed LNI 192.168.204.132@tcp [ 3421.710227] Key type .llcrypt unregistered [ 3421.712630] Key type ._llcrypt unregistered [ 3442.403368] Key type ._llcrypt registered [ 3442.405399] Key type .llcrypt registered [ 3442.529783] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3443.688190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3443.804523] alg: No test for adler32 (adler32-zlib) [ 3444.906514] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 3445.171504] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3446.863338] Key type lgssc registered [ 3448.139445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3456.488090] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3456.528783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.223616] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3477.343532] LustreError: 108884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 3504.032580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.780404] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3518.756180] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3526.522438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.908161] LustreError: MGC192.168.204.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3526.928608] LustreError: 109909:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3526.951467] LustreError: 109909:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3526.964067] LustreError: 109909:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3527.068579] Lustre: server umount lustre-MDT0000 complete [ 3527.071432] LustreError: 109909:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3548.127168] Lustre: 108221:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787388471/real 1787388471] req@00000000717cb8aa x1874212563193344/t0(0) o400->MGC192.168.204.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787388487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3548.127425] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3548.165408] Lustre: 108221:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3548.189428] LustreError: 108896:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3568.703836] Lustre: server umount lustre-OST0000 complete [ 3578.281793] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3589.768181] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3633.528489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.905191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3633.960800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3634.057384] Lustre: lustre-MDT0000: new disk, initializing [ 3634.154552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.182600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3638.831760] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3649.594072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.719362] Lustre: 112706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3649.774459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3649.778438] Lustre: Skipped 1 previous similar message [ 3649.881748] Lustre: lustre-MDT0001: new disk, initializing [ 3649.990593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3650.018894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3650.030113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3655.041348] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3665.314301] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3671.544849] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3678.218749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3678.566563] Lustre: lustre-OST0000: new disk, initializing [ 3678.572887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3678.588068] Lustre: 114674:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3678.724055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.672394] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3685.974269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3685.982321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3686.132326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3695.341973] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3700.098417] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3700.350503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3705.391890] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3705.639785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3711.976673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.982448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.999993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.009214] Lustre: Skipped 1 previous similar message [ 3712.029046] Lustre: Skipped 1 previous similar message [ 3713.975422] Lustre: server umount lustre-OST0000 complete [ 3721.696492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3721.706435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.727168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3725.297752] Lustre: server umount lustre-MDT0000 complete [ 3729.852388] LustreError: 112698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388669 with bad export cookie 12169098427491738334 [ 3729.857482] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.866080] LustreError: 112698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3730.663584] Lustre: server umount lustre-MDT0001 complete [ 3739.493806] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:51:16 (1787388676) [ 3743.359166] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3762.746680] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3773.644926] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3809.408239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.624837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3809.653796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3809.728852] Lustre: lustre-MDT0000: new disk, initializing [ 3809.796119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.816242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3813.682107] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3823.630855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.782964] Lustre: 119577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3823.869574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3823.873416] Lustre: Skipped 1 previous similar message [ 3823.970320] Lustre: lustre-MDT0001: new disk, initializing [ 3824.072898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.129021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3824.137751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3828.018332] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3837.999306] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3843.579727] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3851.920914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.314872] Lustre: lustre-OST0000: new disk, initializing [ 3852.320993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3852.327470] Lustre: 121544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3852.429663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.126992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3861.141882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3861.170327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3861.301468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3871.996289] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.208160] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3877.605091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3883.226873] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3883.492393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3887.086618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.113359] Lustre: Skipped 1 previous similar message [ 3887.129608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.132392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3887.143094] Lustre: Skipped 2 previous similar messages [ 3891.993879] Lustre: server umount lustre-OST0000 complete [ 3897.312302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.323670] Lustre: Skipped 1 previous similar message [ 3897.331741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.341554] Lustre: Skipped 1 previous similar message [ 3902.350137] Lustre: server umount lustre-MDT0000 complete [ 3902.434543] LustreError: 119587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.460234] LustreError: 119587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3906.070259] LustreError: 119569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388845 with bad export cookie 12169098427491739216 [ 3906.083456] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.088016] LustreError: 119569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3906.373955] Lustre: server umount lustre-MDT0001 complete [ 3915.937972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.577983] LustreError: 123761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3916.719987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.892313] LustreError: 123762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.668550] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3927.010264] LustreError: 123761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.143012] LustreError: 123762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.289681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.536916] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3948.328902] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3954.718857] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3961.897795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.939603] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3977.463506] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3994.059196] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4000.225803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.246694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.267281] Lustre: Skipped 1 previous similar message [ 4000.277195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.959504] Lustre: server umount lustre-OST0000 complete [ 4011.489458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4011.506239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.536763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.891626] Lustre: server umount lustre-MDT0000 complete [ 4023.776716] LustreError: 123756:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.814104] LustreError: 123756:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4023.971795] LustreError: 123743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388963 with bad export cookie 12169098427491740098 [ 4023.980102] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.996838] LustreError: 123743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4024.622837] Lustre: server umount lustre-MDT0001 complete [ 4040.060803] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4044.390108] Key type lgssc unregistered [ 4044.904669] LNet: 127416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4044.929837] LNetError: 127416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4044.963327] LNet: Removed LNI 192.168.204.132@tcp [ 4046.144942] Key type .llcrypt unregistered [ 4046.149083] Key type ._llcrypt unregistered [ 4070.592350] Key type ._llcrypt registered [ 4070.594769] Key type .llcrypt registered [ 4070.806402] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 4088.592902] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4090.262390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4090.368953] alg: No test for adler32 (adler32-zlib) [ 4091.648576] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 4091.843851] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4093.559490] Key type lgssc registered [ 4094.925921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4134.426726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4134.465139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4135.656606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4135.691734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4135.829584] Lustre: lustre-MDT0000: new disk, initializing [ 4135.959130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.984251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4140.877694] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4150.834254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.956447] Lustre: 130569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4151.058867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4151.066835] Lustre: Skipped 1 previous similar message [ 4151.165637] Lustre: lustre-MDT0001: new disk, initializing [ 4151.287116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.337069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4151.351322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4155.823934] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4165.148199] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4171.824784] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4178.761656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.039578] Lustre: lustre-OST0000: new disk, initializing [ 4179.044794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4179.050817] Lustre: 132539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4179.125604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.862180] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4186.200806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4186.212862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4186.280831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4195.372297] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4201.426090] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4201.674562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4206.235624] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4206.408376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4212.193871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.194346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4212.226166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.246300] Lustre: Skipped 1 previous similar message [ 4214.747972] Lustre: server umount lustre-OST0000 complete [ 4222.433811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.440805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.452987] Lustre: Skipped 2 previous similar messages [ 4225.392503] Lustre: server umount lustre-MDT0000 complete [ 4229.462792] LustreError: 130562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389169 with bad export cookie 12692423636699038685 [ 4229.475484] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4229.476162] LustreError: 130562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4230.187826] Lustre: server umount lustre-MDT0001 complete [ 4241.198538] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 04:59:38 (1787389178) [ 4248.750934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.293541] LustreError: 134948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.311629] LustreError: 134948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4249.401375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.052055] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4254.692563] LustreError: 134949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.811509] LustreError: 134948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.461980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.931146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4268.665322] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4278.322553] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4285.085340] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4289.504287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4289.506596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.521238] LustreError: Skipped 1 previous similar message [ 4289.543353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4289.561146] Lustre: Skipped 2 previous similar messages [ 4293.780946] Lustre: server umount lustre-MDT0000 complete [ 4294.625548] LustreError: 134943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.651576] LustreError: 134943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4297.872126] LustreError: 134931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389237 with bad export cookie 12692423636699039567 [ 4297.879902] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4297.884571] LustreError: 134931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4298.264755] Lustre: server umount lustre-MDT0001 complete [ 4307.528288] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:00:45 (1787389245) [ 4313.888119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4329.503693] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4334.623866] LustreError: 137364:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 4360.351665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.351278] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4374.584919] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4376.352858] Lustre: server umount lustre-OST0000 complete [ 4384.566729] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:02:02 (1787389322) [ 4391.271659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.708762] LustreError: 138821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.858513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.744368] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4402.146556] LustreError: 138822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.190146] LustreError: 138822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4407.196329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.811853] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4421.073598] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4427.379468] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4434.010915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.410891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4434.425027] Lustre: Skipped 1 previous similar message [ 4440.559840] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4448.632491] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4462.076706] Lustre: server umount lustre-MDT0000 complete [ 4463.077836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.079647] LustreError: 138818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.094591] Lustre: Skipped 2 previous similar messages [ 4463.109893] LustreError: 138818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4466.199510] LustreError: 138802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389406 with bad export cookie 12692423636699040232 [ 4466.231737] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4466.537798] Lustre: server umount lustre-MDT0001 complete [ 4480.936712] Lustre: server umount lustre-OST0000 complete [ 4489.284658] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:03:47 (1787389427) [ 4495.704883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.122584] LustreError: 142496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4496.143074] LustreError: 142496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4500.444418] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4508.999591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.414071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.422553] Lustre: Skipped 1 previous similar message [ 4513.670982] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4521.165527] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4527.233827] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4533.884754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4540.414856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4541.099471] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4550.724690] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4556.277895] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4556.620768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4563.281037] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4563.528862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4565.990454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.995788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4566.015884] Lustre: Skipped 2 previous similar messages [ 4566.020221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4566.032082] Lustre: Skipped 1 previous similar message [ 4571.128216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.135360] Lustre: Skipped 1 previous similar message [ 4580.319395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4580.509545] Lustre: server umount lustre-OST0000 complete [ 4586.469086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.472038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.489418] Lustre: Skipped 1 previous similar message [ 4586.492214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.496234] Lustre: Skipped 3 previous similar messages [ 4599.263147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.529888] Lustre: server umount lustre-MDT0000 complete [ 4601.824740] LustreError: 144627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.849778] LustreError: 144627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4603.160582] LustreError: 143230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389543 with bad export cookie 12692423636699041296 [ 4603.166217] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.178600] LustreError: 143230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4603.497963] Lustre: server umount lustre-MDT0001 complete [ 4612.571976] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:05:49 (1787389549) [ 4619.920636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4635.551898] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4640.671886] LustreError: 146657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 4666.399422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.407554] Lustre: Skipped 1 previous similar message [ 4672.812487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4680.603352] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4687.469690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.111662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4692.704599] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4701.456943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.699835] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4712.655487] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4718.471408] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4723.470727] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4723.766427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4728.406174] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4728.655726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4732.385690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4732.399446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4732.413196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4732.417414] Lustre: Skipped 4 previous similar messages [ 4736.856796] Lustre: server umount lustre-MDT0000 complete [ 4738.029144] LustreError: 147695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.063617] LustreError: 147695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4740.403708] LustreError: 146663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389680 with bad export cookie 12692423636699042325 [ 4740.411739] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.421205] LustreError: 146663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4740.731340] Lustre: server umount lustre-MDT0001 complete [ 4754.706735] Lustre: server umount lustre-OST0000 complete [ 4763.597542] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:08:21 (1787389701) [ 4770.838236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4786.463745] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4791.583443] LustreError: 150851:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 4824.416268] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4832.317562] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4839.965951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.668571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4845.677126] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4854.718738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.810373] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4867.483382] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4873.145589] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4881.227910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4881.420715] Lustre: lustre-OST0001: new disk, initializing [ 4881.423758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4881.428728] Lustre: 153722:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4888.422446] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4889.173116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4889.197414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4889.296307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4898.792469] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4903.939243] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4904.200947] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4909.364996] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4909.550695] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4911.587349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4911.595323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.608777] Lustre: Skipped 2 previous similar messages [ 4911.618439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.621865] Lustre: Skipped 2 previous similar messages [ 4917.899533] Lustre: server umount lustre-OST0000 complete [ 4927.483120] Lustre: server umount lustre-OST0001 complete [ 4932.067354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4932.080308] LustreError: Skipped 3 previous similar messages [ 4937.516499] Lustre: server umount lustre-MDT0000 complete [ 4941.967268] LustreError: 150857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389881 with bad export cookie 12692423636699043277 [ 4941.987799] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4941.990774] LustreError: 150857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4942.489389] Lustre: server umount lustre-MDT0001 complete [ 4965.869251] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:11:43 (1787389903) [ 4967.732699] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4969.325811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4970.771159] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:11:48 (1787389908) [ 4978.133829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.465204] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4978.514620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4978.730829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.739147] Lustre: Skipped 6 previous similar messages [ 4983.646155] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4995.722782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.921147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4995.970273] Lustre: 157286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5001.905178] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5010.004875] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5015.745541] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5022.406401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5022.524650] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5028.939214] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5031.665143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5037.479189] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5043.879269] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5044.177481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5049.151729] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5049.467232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5052.384531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.385468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.406535] LustreError: Skipped 1 previous similar message [ 5052.408984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5052.408990] Lustre: Skipped 8 previous similar messages [ 5052.469625] Lustre: Skipped 6 previous similar messages [ 5057.619988] Lustre: server umount lustre-OST0000 complete [ 5078.848882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5082.353805] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.370995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.374312] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5082.393690] Lustre: Skipped 1 previous similar message [ 5085.272253] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5094.487954] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5101.328510] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5101.581292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5108.484902] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5108.832465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5115.248775] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b17c5a48000.ost_server_uuid 50 [ 5117.181780] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b17c5a48000.ost_server_uuid in FULL state after 0 sec [ 5117.371540] LustreError: 157294:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.32@tcp) returned error from blocking AST (req@00000000eafa0623 x1874213240840064 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f39156e7/0xb0248c0cdc37df27 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.32@tcp remote: 0xdee30b9de831f142 expref: 5 pid: 157295 timeout: 5217 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5117.426321] LustreError: lustre-MDT0000: A client on nid 192.168.204.32@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5117.448797] LustreError: 157283:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.32@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f39156e7/0xb0248c0cdc37df27 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.32@tcp remote: 0xdee30b9de831f142 expref: 6 pid: 157295 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5123.043579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5127.472149] Lustre: server umount lustre-OST0000 complete [ 5137.789301] Lustre: server umount lustre-MDT0000 complete [ 5138.912647] LustreError: 158237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.940702] LustreError: 158237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5141.942690] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390081 with bad export cookie 12692423636699044453 [ 5141.958285] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5141.967582] LustreError: 157279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5156.324399] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5156.720970] Lustre: server umount lustre-MDT0001 complete [ 5168.292828] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5171.549355] Key type lgssc unregistered [ 5171.874820] LNet: 163404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5171.890531] LNetError: 163404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5171.917120] LNet: Removed LNI 192.168.204.132@tcp [ 5172.895668] Key type .llcrypt unregistered [ 5172.898674] Key type ._llcrypt unregistered [ 5201.738165] Key type ._llcrypt registered [ 5201.741643] Key type .llcrypt registered [ 5201.840546] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5202.903698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5203.020226] alg: No test for adler32 (adler32-zlib) [ 5204.017565] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5204.225674] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5205.919173] Key type lgssc registered [ 5206.865560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5215.362238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5215.389507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.828048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.031392] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5226.989620] LustreError: 164818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5227.017633] LustreError: 164818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5231.546904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.096498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.766487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5244.766254] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.006698] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5259.153820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.500909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5261.562446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5266.324925] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5273.480084] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5283.361325] Lustre: Failing over lustre-MDT0000 [ 5283.596428] Lustre: server umount lustre-MDT0000 complete [ 5287.397746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.402954] LustreError: 166006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.422061] Lustre: Skipped 2 previous similar messages [ 5287.455354] LustreError: 166006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5292.513685] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5292.525581] LustreError: 164812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5296.008052] LustreError: 165553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.32@tcp arrived at 1787390235 with bad export cookie 5373597652874752575 [ 5296.021564] LustreError: 165553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5297.633991] LustreError: 164817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.674201] LustreError: 164817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5302.754724] LustreError: 165546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.779500] LustreError: 165546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5302.928175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.018976] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.304301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.375518] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5307.767263] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5308.390559] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5308.396476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5310.320924] Lustre: lustre-MDT0000: Denying connection for new client 2835a28c-cda4-4879-8fbe-4e6b5bcd0bcb (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5323.237641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5323.255355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.280008] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5328.867206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5328.879357] Lustre: Skipped 2 previous similar messages [ 5332.960496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5333.258266] Lustre: server umount lustre-OST0000 complete [ 5337.431091] LustreError: 168550:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5337.448335] Lustre: 167899:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5337.456343] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5337.498174] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5337.506955] LustreError: 167899:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000583e1f24 x1874214407682432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5337.522107] LustreError: 167899:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5337.537282] LustreError: 167899:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5337.551241] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5338.085905] Lustre: server umount lustre-MDT0000 complete [ 5339.107834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.140068] Lustre: Skipped 1 previous similar message [ 5339.167735] LustreError: 166006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5343.300299] LustreError: 165553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390283 with bad export cookie 5373597652874752680 [ 5343.309924] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5343.314528] LustreError: 165553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5349.820479] Lustre: server umount lustre-MDT0001 complete [ 5360.760742] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5363.441964] Key type lgssc unregistered [ 5363.777883] LNet: 169302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5363.785030] LNetError: 169302:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5364.837960] LNet: Removed LNI 192.168.204.132@tcp [ 5366.048361] Key type .llcrypt unregistered [ 5366.051853] Key type ._llcrypt unregistered [ 5386.852097] Key type ._llcrypt registered [ 5386.855853] Key type .llcrypt registered [ 5386.939828] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5388.292651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5388.331056] alg: No test for adler32 (adler32-zlib) [ 5389.369164] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5389.639493] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5391.335188] Key type lgssc registered [ 5392.659385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5401.819874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5401.841595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.360707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.294886] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5413.858956] LustreError: 170696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.888481] LustreError: 170696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5418.987710] LustreError: 170697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.935567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.390539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5424.207743] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5431.296071] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5437.561175] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5444.532327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5444.877843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5449.040740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5451.968966] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5460.647917] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5469.665076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.672193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.691863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5470.729175] Lustre: server umount lustre-OST0000 complete [ 5476.832608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5476.844591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.852530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5477.863484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.866650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5480.933618] Lustre: server umount lustre-MDT0000 complete [ 5482.980549] LustreError: 170696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.005731] LustreError: 170696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5485.260294] LustreError: 170677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390425 with bad export cookie 6172376954166627437 [ 5485.273621] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5495.923264] Lustre: server umount lustre-MDT0001 complete [ 5506.133097] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5508.755725] Key type lgssc unregistered [ 5509.004985] LNet: 174230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5509.010493] LNetError: 174230:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5509.033874] LNet: Removed LNI 192.168.204.132@tcp [ 5509.916394] Key type .llcrypt unregistered [ 5509.924077] Key type ._llcrypt unregistered [ 5542.184751] Key type ._llcrypt registered [ 5542.190419] Key type .llcrypt registered [ 5542.384457] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5544.315387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5544.366209] alg: No test for adler32 (adler32-zlib) [ 5545.684519] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5546.061360] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5548.015171] Key type lgssc registered [ 5550.513546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5561.872718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5561.899805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.394546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.100599] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5573.607799] LustreError: 175933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.631054] LustreError: 175933:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5577.891506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.290729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5583.276256] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5590.549503] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5596.144921] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5602.860867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.204455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.442219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5610.124756] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5619.036696] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5628.905631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5628.918363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.919244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5628.948023] Lustre: Skipped 1 previous similar message [ 5633.467494] Lustre: server umount lustre-OST0000 complete [ 5639.142400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.146150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5639.162241] Lustre: Skipped 1 previous similar message [ 5639.185552] Lustre: Skipped 2 previous similar messages [ 5643.801843] Lustre: server umount lustre-MDT0000 complete [ 5644.269146] LustreError: 175929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.312093] LustreError: 175929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5648.353297] LustreError: 177523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5648.365926] LustreError: 177523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5648.458697] LustreError: 175913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390588 with bad export cookie 9078658001858717384 [ 5648.465708] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.475235] LustreError: 175913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5648.970681] Lustre: server umount lustre-MDT0001 complete [ 5661.800514] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5664.600602] Key type lgssc unregistered [ 5664.858150] LNet: 179469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5664.866172] LNetError: 179469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5664.882983] LNet: Removed LNI 192.168.204.132@tcp [ 5665.607318] Key type .llcrypt unregistered [ 5665.610746] Key type ._llcrypt unregistered [ 5691.757257] Key type ._llcrypt registered [ 5691.760536] Key type .llcrypt registered [ 5691.891814] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5693.132452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5693.148316] alg: No test for adler32 (adler32-zlib) [ 5694.391967] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5694.615589] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5696.295183] Key type lgssc registered [ 5697.649647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5709.291746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5709.306579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.780931] Lustre: *** cfs_fail_loc=135, val=0*** [ 5710.844529] LustreError: 181007:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5710.849386] LustreError: 181007:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -2 [ 5710.854818] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5710.854818] [ 5710.860163] LustreError: MGC192.168.204.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5710.867421] LustreError: 180969:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5710.871727] LustreError: 180969:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5710.878913] LustreError: 180969:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5710.882511] LustreError: 180969:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5710.968899] Lustre: server umount lustre-MDT0000 complete [ 5710.974558] LustreError: 180969:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5718.928168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.380680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.308357] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5729.761094] LustreError: 181379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.795862] LustreError: 181379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5734.030631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.592068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5739.356683] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5747.829492] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.392886] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5759.976618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5759.984109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.010802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.485208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.572156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5761.577535] Lustre: Skipped 3 previous similar messages [ 5763.706798] Lustre: server umount lustre-MDT0000 complete [ 5766.629641] LustreError: 181380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.654057] LustreError: 181380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5767.762121] LustreError: 181360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390707 with bad export cookie 12133863713529763453 [ 5767.777288] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5767.788100] LustreError: 181360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5768.169428] Lustre: server umount lustre-MDT0001 complete [ 5781.230192] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5784.103144] Key type lgssc unregistered [ 5784.393861] LNet: 183702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5784.399252] LNetError: 183702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5784.422351] LNet: Removed LNI 192.168.204.132@tcp [ 5785.188157] Key type .llcrypt unregistered [ 5785.192946] Key type ._llcrypt unregistered [ 5808.961216] Key type ._llcrypt registered [ 5808.965273] Key type .llcrypt registered [ 5809.045910] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5809.332521] Key type .llcrypt unregistered [ 5809.351467] Key type ._llcrypt unregistered [ 5828.425814] Key type ._llcrypt registered [ 5828.428276] Key type .llcrypt registered [ 5828.610537] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5830.231533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5830.358838] alg: No test for adler32 (adler32-zlib) [ 5831.462353] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 5831.712819] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5833.415218] Key type lgssc registered [ 5834.563705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5842.388077] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5842.404229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.951610] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5863.071853] LustreError: 185840:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 5889.759274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.294168] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5903.553724] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5910.870415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.281273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.344413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5916.203548] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5916.642477] LustreError: 185866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.660633] LustreError: 185866:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5921.765297] LustreError: 185865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.841315] LustreError: 186876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.860747] LustreError: 186876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5925.436423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.846337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.645946] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5939.761292] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5946.192310] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5949.770255] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5962.721759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.731973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.758906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.826717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.827180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.857208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.939593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.948924] Lustre: Skipped 2 previous similar messages [ 5976.031190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5976.217844] Lustre: server umount lustre-OST0000 complete [ 5982.182159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.184124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.191419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5982.197265] Lustre: Skipped 1 previous similar message [ 5986.397958] Lustre: server umount lustre-MDT0000 complete [ 5987.297939] LustreError: 188037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.326670] LustreError: 188037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5990.370509] LustreError: 185847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390930 with bad export cookie 4408670779064185079 [ 5990.388120] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.388585] LustreError: 185847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5990.802668] Lustre: server umount lustre-MDT0001 complete [ 6003.726487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6006.561902] Key type lgssc unregistered [ 6006.813060] LNet: 189811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6006.827348] LNetError: 189811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6006.863340] LNet: Removed LNI 192.168.204.132@tcp [ 6007.578331] Key type .llcrypt unregistered [ 6007.581283] Key type ._llcrypt unregistered [ 6032.081984] Key type ._llcrypt registered [ 6032.084994] Key type .llcrypt registered [ 6032.188766] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6033.467747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6033.513491] alg: No test for adler32 (adler32-zlib) [ 6034.649722] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6034.964379] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6036.655413] Key type lgssc registered [ 6037.901842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.616137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6047.632263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.139153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.708393] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6058.472668] LustreError: 191239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.526488] LustreError: 191239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6063.592073] LustreError: 191238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.934298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.212956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6071.945768] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6080.793572] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6087.266913] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6094.469727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.879929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.422201] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6108.648773] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6118.460054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6124.059911] Lustre: Failing over lustre-MDT0000 [ 6124.369695] Lustre: server umount lustre-MDT0000 complete [ 6125.594576] LustreError: 193083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6127.586235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6127.598204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.626982] LustreError: 191238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.614406] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.644276] Lustre: Skipped 1 previous similar message [ 6133.736685] LustreError: 191233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.775656] LustreError: 191233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6143.969366] LustreError: 193083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.986644] LustreError: 193083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6144.422160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.594594] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6144.863209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6144.920573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6146.045967] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6149.378111] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6150.126700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6150.148158] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6150.212099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6154.133859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6162.393093] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6175.205519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6175.211611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.234339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6176.205352] Lustre: server umount lustre-OST0000 complete [ 6180.833029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.846097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.851935] Lustre: Skipped 2 previous similar messages [ 6180.861138] Lustre: Skipped 1 previous similar message [ 6185.952455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6185.964578] Lustre: Skipped 1 previous similar message [ 6186.960587] Lustre: server umount lustre-MDT0000 complete [ 6191.073179] LustreError: 192178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.091994] LustreError: 192178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6191.174715] LustreError: 193086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391131 with bad export cookie 12122513742656805650 [ 6191.185062] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6191.188585] LustreError: 193086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6191.559922] Lustre: server umount lustre-MDT0001 complete [ 6201.619192] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6204.167507] Key type lgssc unregistered [ 6204.461871] LNet: 196153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6204.467938] LNetError: 196153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6204.490730] LNet: Removed LNI 192.168.204.132@tcp [ 6205.331695] Key type .llcrypt unregistered [ 6205.334199] Key type ._llcrypt unregistered [ 6228.757678] Key type ._llcrypt registered [ 6228.759779] Key type .llcrypt registered [ 6228.863950] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6229.953953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6230.080191] alg: No test for adler32 (adler32-zlib) [ 6231.063834] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6231.302151] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6232.961655] Key type lgssc registered [ 6233.991376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6241.759730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6241.778776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.205424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.504480] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6252.519060] LustreError: 197565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6252.535788] LustreError: 197565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6257.640190] LustreError: 197564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.644484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6259.225049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6264.073614] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6270.501733] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6276.212107] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6283.153346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6283.553127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.679694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6289.301198] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6296.219577] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6299.044305] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6306.753972] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6325.353297] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6339.041498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6339.049077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.051493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6339.070437] Lustre: Skipped 1 previous similar message [ 6344.169773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6344.183044] Lustre: Skipped 2 previous similar messages [ 6345.062439] Lustre: server umount lustre-OST0000 complete [ 6349.291958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.304788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.317621] Lustre: Skipped 1 previous similar message [ 6349.329862] Lustre: Skipped 1 previous similar message [ 6354.400727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6354.407401] Lustre: Skipped 1 previous similar message [ 6359.521516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6359.528018] Lustre: Skipped 1 previous similar message [ 6363.103205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6363.377870] Lustre: server umount lustre-MDT0000 complete [ 6364.640773] LustreError: 197565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6364.657413] LustreError: 197565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6366.969664] LustreError: 199411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391306 with bad export cookie 4843648660929082360 [ 6366.992729] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6367.286388] Lustre: server umount lustre-MDT0001 complete [ 6379.299627] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6381.940240] Key type lgssc unregistered [ 6382.262345] LNet: 201460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6382.274985] LNetError: 201460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6382.315433] LNet: Removed LNI 192.168.204.132@tcp [ 6383.248686] Key type .llcrypt unregistered [ 6383.254403] Key type ._llcrypt unregistered [ 6402.856603] Key type ._llcrypt registered [ 6402.858759] Key type .llcrypt registered [ 6403.002280] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6404.283343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6404.353278] alg: No test for adler32 (adler32-zlib) [ 6405.427662] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6405.701597] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6407.383190] Key type lgssc registered [ 6408.591946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6416.317366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6416.356832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.912935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.828527] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6428.133760] LustreError: 202728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.165290] LustreError: 202728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6431.655928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.965348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6436.811382] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6444.175751] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6449.477918] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6457.548549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6458.026705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6463.165882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6464.436225] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6472.736786] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6479.843514] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6489.468173] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6500.510052] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6506.802336] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6517.745793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6517.745891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.770772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6517.781429] Lustre: Skipped 1 previous similar message [ 6517.796392] Lustre: Skipped 1 previous similar message [ 6521.857081] Lustre: server umount lustre-OST0000 complete [ 6527.970953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.994414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6528.006220] Lustre: Skipped 1 previous similar message [ 6533.089870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6533.095770] Lustre: Skipped 1 previous similar message [ 6533.598657] Lustre: server umount lustre-MDT0000 complete [ 6537.193654] LustreError: 203455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.218273] LustreError: 203455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6537.300500] LustreError: 202709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391477 with bad export cookie 6085562404793611103 [ 6537.308709] LustreError: 202709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6537.309471] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6537.633972] Lustre: server umount lustre-MDT0001 complete [ 6550.894953] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6555.188621] Key type lgssc unregistered [ 6555.579740] LNet: 207555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6555.594405] LNetError: 207555:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6555.635547] LNet: Removed LNI 192.168.204.132@tcp [ 6556.637129] Key type .llcrypt unregistered [ 6556.641134] Key type ._llcrypt unregistered [ 6586.366469] Key type ._llcrypt registered [ 6586.371677] Key type .llcrypt registered [ 6586.565513] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6587.813583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6587.987553] alg: No test for adler32 (adler32-zlib) [ 6589.221880] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6589.432536] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6591.088979] Key type lgssc registered [ 6592.250134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6602.206511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6602.258578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.914518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.750475] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6613.483897] LustreError: 208972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.512409] LustreError: 208972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6618.604476] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.538968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.049596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.104106] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6635.349828] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6642.193530] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6649.836592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.298423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.242516] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6663.767396] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6673.869162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6689.260534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6689.274956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6689.290647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6689.296889] Lustre: Skipped 1 previous similar message [ 6694.372449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.384480] Lustre: Skipped 1 previous similar message [ 6695.102241] Lustre: server umount lustre-OST0000 complete [ 6699.493560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6699.515087] Lustre: Skipped 2 previous similar messages [ 6699.522498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.609239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.615896] Lustre: Skipped 2 previous similar messages [ 6706.150292] Lustre: server umount lustre-MDT0000 complete [ 6709.734844] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.760314] LustreError: 208971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6710.685195] LustreError: 208952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391650 with bad export cookie 4032224909188527878 [ 6710.696717] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6710.706349] LustreError: 208952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6711.178545] Lustre: server umount lustre-MDT0001 complete [ 6723.086190] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6725.866418] Key type lgssc unregistered [ 6726.177057] LNet: 212676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6726.183532] LNetError: 212676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6726.208924] LNet: Removed LNI 192.168.204.132@tcp [ 6727.140839] Key type .llcrypt unregistered [ 6727.145103] Key type ._llcrypt unregistered [ 6750.874801] Key type ._llcrypt registered [ 6750.877423] Key type .llcrypt registered [ 6750.958571] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6752.537031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6752.557691] alg: No test for adler32 (adler32-zlib) [ 6753.744512] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6754.097464] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6755.871440] Key type lgssc registered [ 6756.984225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6765.455422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6765.497892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.088260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.263484] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6776.311578] LustreError: 213947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.331268] LustreError: 213947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6780.572484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6780.883070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6785.655569] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6794.142820] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6800.370228] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6807.497572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.821530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6812.986227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6814.486751] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6822.399481] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6840.804324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6840.808235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.815397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6840.821535] Lustre: Skipped 1 previous similar message [ 6840.832151] Lustre: Skipped 1 previous similar message [ 6845.852653] Lustre: server umount lustre-OST0000 complete [ 6851.041042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.053303] Lustre: Skipped 1 previous similar message [ 6851.063065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.081381] Lustre: Skipped 1 previous similar message [ 6855.928959] Lustre: server umount lustre-MDT0000 complete [ 6856.172785] LustreError: 214674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.207539] LustreError: 214674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6859.997790] LustreError: 213927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391799 with bad export cookie 17774304801239991657 [ 6860.000194] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.016179] LustreError: 213927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6860.387413] Lustre: server umount lustre-MDT0001 complete [ 6871.556432] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6874.607347] Key type lgssc unregistered [ 6875.056497] LNet: 217632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6875.071171] LNetError: 217632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6875.094613] LNet: Removed LNI 192.168.204.132@tcp [ 6876.135946] Key type .llcrypt unregistered [ 6876.137813] Key type ._llcrypt unregistered [ 6896.418663] Key type ._llcrypt registered [ 6896.420779] Key type .llcrypt registered [ 6896.511407] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6897.581139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6897.651597] alg: No test for adler32 (adler32-zlib) [ 6898.879630] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 6899.179365] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6900.943177] Key type lgssc registered [ 6902.566309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6911.546975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6911.607187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.302098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.766923] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6923.748630] LustreError: 218899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6923.768904] LustreError: 218899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6928.278993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.676610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6933.818343] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6943.113692] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6949.550985] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6957.440690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.766953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6958.838500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6964.317911] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6973.687957] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6984.677331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.764796] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6985.052678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.561684] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7002.014798] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7003.715867] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7015.985108] Lustre: Permanently deactivating lustre-OST0001 [ 7016.006817] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7025.283729] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7034.971469] Lustre: Permanently reactivating lustre-OST0001 [ 7034.989404] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7034.996704] Lustre: Skipped 2 previous similar messages [ 7042.635626] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.656291] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7042.661209] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7042.673048] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7042.708659] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7048.161113] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7048.172106] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.173167] LustreError: Skipped 1 previous similar message [ 7048.174788] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7048.214153] Lustre: Skipped 2 previous similar messages [ 7049.706899] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7049.726828] Lustre: Skipped 2 previous similar messages [ 7051.915654] Lustre: server umount lustre-OST0001 complete [ 7054.824570] LustreError: 220721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7054.841515] LustreError: 220721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7059.937045] LustreError: 221746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7059.947448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.953279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7059.958589] LustreError: 221746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7059.968909] Lustre: Skipped 1 previous similar message [ 7062.872580] Lustre: server umount lustre-OST0000 complete [ 7067.104501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7067.111382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.135180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7067.142474] Lustre: Skipped 1 previous similar message [ 7073.259627] Lustre: server umount lustre-MDT0000 complete [ 7075.296535] LustreError: 219627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.317912] LustreError: 219627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7077.333556] LustreError: 218882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392017 with bad export cookie 17630718129425914463 [ 7077.341246] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.347775] LustreError: 218882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7077.835155] Lustre: server umount lustre-MDT0001 complete [ 7091.751432] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7095.640900] Key type lgssc unregistered [ 7095.992878] LNet: 223973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7095.999206] LNetError: 223973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7096.014354] LNet: Removed LNI 192.168.204.132@tcp [ 7097.089405] Key type .llcrypt unregistered [ 7097.091727] Key type ._llcrypt unregistered [ 7135.159863] Key type ._llcrypt registered [ 7135.164539] Key type .llcrypt registered [ 7135.249867] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7135.891144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7135.903120] alg: No test for adler32 (adler32-zlib) [ 7137.037632] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7137.234052] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7138.881838] Key type lgssc registered [ 7139.965140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7147.108644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7147.127352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.285044] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7148.303917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7148.412055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.379745] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7161.822233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.936190] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7161.957746] Lustre: 226173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7162.203099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7166.845823] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7174.615549] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7180.309042] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7187.513121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7187.707167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7187.880864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.976639] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7197.923167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7202.496825] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7206.462064] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7213.289397] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7220.283121] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7227.324885] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7236.518207] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7254.425920] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7254.432340] Lustre: Skipped 1 previous similar message [ 7295.168175] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7295.180952] Lustre: Skipped 4 previous similar messages [ 7362.482289] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7362.487744] Lustre: Skipped 7 previous similar messages [ 7392.739634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7392.743446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.754845] LustreError: Skipped 1 previous similar message [ 7392.763875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7392.779732] Lustre: Skipped 1 previous similar message [ 7392.798236] Lustre: Skipped 1 previous similar message [ 7394.468598] Lustre: server umount lustre-OST0000 complete [ 7402.978103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.978922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.996858] Lustre: Skipped 1 previous similar message [ 7403.015254] Lustre: Skipped 1 previous similar message [ 7404.584933] Lustre: server umount lustre-MDT0000 complete [ 7408.300590] LustreError: 229234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392348 with bad export cookie 492391823773008003 [ 7408.309926] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7408.313397] LustreError: 229234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7408.713856] Lustre: server umount lustre-MDT0001 complete [ 7419.187224] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7422.309828] Key type lgssc unregistered [ 7422.613377] LNet: 230987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7422.626188] LNetError: 230987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7422.643636] LNet: Removed LNI 192.168.204.132@tcp [ 7423.407066] Key type .llcrypt unregistered [ 7423.412418] Key type ._llcrypt unregistered [ 7447.572593] Key type ._llcrypt registered [ 7447.574996] Key type .llcrypt registered [ 7447.664776] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7448.800658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7448.839661] alg: No test for adler32 (adler32-zlib) [ 7449.835576] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7450.099170] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7451.807537] Key type lgssc registered [ 7453.338457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7461.827986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7461.864643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.323446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.022063] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7473.637945] LustreError: 232402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.652525] LustreError: 232402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7475.995052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.354403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.563077] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7491.094687] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7497.064358] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7504.021868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.417372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7508.534492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7510.370745] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7518.048278] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7537.121178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7537.136393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.177252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7539.175541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7539.195530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.207393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7542.092527] Lustre: server umount lustre-OST0000 complete [ 7547.880419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7547.895544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.919305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.933773] Lustre: Skipped 2 previous similar messages [ 7550.439667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7550.461229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.184246] Lustre: server umount lustre-MDT0000 complete [ 7555.552846] LustreError: 232397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.571893] LustreError: 232397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7557.325694] LustreError: 232384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392497 with bad export cookie 6071783360275743846 [ 7557.339157] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7557.346804] LustreError: 232384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7557.725757] Lustre: server umount lustre-MDT0001 complete [ 7570.082837] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7573.104481] Key type lgssc unregistered [ 7573.414797] LNet: 236090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7573.430653] LNetError: 236090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7573.450497] LNet: Removed LNI 192.168.204.132@tcp [ 7574.210760] Key type .llcrypt unregistered [ 7574.213176] Key type ._llcrypt unregistered [ 7598.738648] Key type ._llcrypt registered [ 7598.741465] Key type .llcrypt registered [ 7598.879314] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7599.789090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7599.829617] alg: No test for adler32 (adler32-zlib) [ 7600.866584] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7601.066921] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7602.767221] Key type lgssc registered [ 7603.808381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7873.295935] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7875.921554] Key type lgssc unregistered [ 7876.182860] LNet: 238111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7876.187351] LNetError: 238111:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7876.203358] LNet: Removed LNI 192.168.204.132@tcp [ 7876.909641] Key type .llcrypt unregistered [ 7876.913226] Key type ._llcrypt unregistered [ 7903.398978] Key type ._llcrypt registered [ 7903.403240] Key type .llcrypt registered [ 7903.512843] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 7919.435186] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7920.260363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7920.315260] alg: No test for adler32 (adler32-zlib) [ 7921.385757] Lustre: Lustre: Build Version: 2.17.54_112_gd76e9af [ 7921.578753] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7923.224646] Key type lgssc registered [ 7924.326763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7955.490693] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7701 sec ========== 06:01:32 (1787392892) [ 7957.283779] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:01:34 (1787392894) === [ 7960.869424] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:01:38 (1787392898) === [ 7998.746777] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 8001.082628] Key type lgssc unregistered [ 8001.379778] LNet: 243359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8001.393781] LNetError: 243359:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8001.420447] LNet: Removed LNI 192.168.204.132@tcp [ 8002.338156] Key type .llcrypt unregistered [ 8002.342596] Key type ._llcrypt unregistered