[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 591143040 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.008430] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009094] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011361] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.026502] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032802] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036123] *** VALIDATE tmpfs *** [ 0.038514] *** VALIDATE proc *** [ 0.039283] *** VALIDATE cgroup *** [ 0.040013] *** VALIDATE cgroup2 *** [ 0.041457] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045025] Spectre V2 : User space: Vulnerable [ 0.046010] Speculative Store Bypass: Vulnerable [ 0.049697] debug: unmapping init [mem 0xffffffff8ea59000-0xffffffff8ea60fff] [ 0.052461] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053874] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055028] ... version: 2 [ 0.056014] ... bit width: 48 [ 0.057012] ... generic registers: 4 [ 0.058013] ... value mask: 0000ffffffffffff [ 0.059137] ... max period: 00007fffffffffff [ 0.060019] ... fixed-purpose events: 3 [ 0.061015] ... event mask: 000000070000000f [ 0.062515] rcu: Hierarchical SRCU implementation. [ 0.064818] smp: Bringing up secondary CPUs ... [ 0.065825] x86: Booting SMP configuration: [ 0.066031] .... node #0, CPUs: #1 #2 #3 [ 0.076252] smp: Brought up 1 node, 4 CPUs [ 0.078017] smpboot: Max logical packages: 1 [ 0.079016] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.125238] node 0 deferred pages initialised in 44ms [ 0.129410] devtmpfs: initialized [ 0.130471] x86/mm: Memory block size: 128MB [ 0.133839] gcov: version magic: 0x41383552 [ 0.135317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137334] pinctrl core: initialized pinctrl subsystem [ 0.138211] [ 0.139010] ************************************************************* [ 0.140015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141015] ** ** [ 0.142013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143012] ** ** [ 0.144015] ** This means that this kernel is built to expose internal ** [ 0.145013] ** IOMMU data structures, which may compromise security on ** [ 0.146137] ** your system. ** [ 0.147013] ** ** [ 0.148014] ** If you see this message and you are not debugging the ** [ 0.149013] ** kernel, report this immediately to your vendor! ** [ 0.150013] ** ** [ 0.151017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152026] ************************************************************* [ 0.153935] NET: Registered protocol family 16 [ 0.154442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.156128] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.158178] cpuidle: using governor menu [ 0.161247] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165821] PCI: Using configuration type 1 for base access [ 0.168288] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.179207] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186640] cryptd: max_cpu_qlen set to 1000 [ 0.188278] ACPI: Added _OSI(Module Device) [ 0.189000] ACPI: Added _OSI(Processor Device) [ 0.190016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192075] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200753] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210027] ACPI: Interpreter enabled [ 0.211057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212012] ACPI: Using IOAPIC for interrupt routing [ 0.213137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214578] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234475] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237167] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240802] acpiphp: Slot [2] registered [ 0.241168] acpiphp: Slot [5] registered [ 0.242156] acpiphp: Slot [6] registered [ 0.243181] acpiphp: Slot [7] registered [ 0.244157] acpiphp: Slot [8] registered [ 0.245141] acpiphp: Slot [9] registered [ 0.246158] acpiphp: Slot [10] registered [ 0.247147] acpiphp: Slot [3] registered [ 0.248164] acpiphp: Slot [4] registered [ 0.250135] acpiphp: Slot [11] registered [ 0.252159] acpiphp: Slot [12] registered [ 0.253120] acpiphp: Slot [13] registered [ 0.254117] acpiphp: Slot [14] registered [ 0.255142] acpiphp: Slot [15] registered [ 0.256149] acpiphp: Slot [16] registered [ 0.257127] acpiphp: Slot [17] registered [ 0.258313] acpiphp: Slot [18] registered [ 0.259130] acpiphp: Slot [19] registered [ 0.260140] acpiphp: Slot [20] registered [ 0.261140] acpiphp: Slot [21] registered [ 0.262451] acpiphp: Slot [22] registered [ 0.263136] acpiphp: Slot [23] registered [ 0.264449] acpiphp: Slot [24] registered [ 0.265131] acpiphp: Slot [25] registered [ 0.266971] acpiphp: Slot [26] registered [ 0.267230] acpiphp: Slot [27] registered [ 0.268115] acpiphp: Slot [28] registered [ 0.269121] acpiphp: Slot [29] registered [ 0.270119] acpiphp: Slot [30] registered [ 0.271132] acpiphp: Slot [31] registered [ 0.272060] PCI host bridge to bus 0000:00 [ 0.273021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.275022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.276025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.278025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280990] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.282344] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.288014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.291048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.292020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.293020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.294025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.295706] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.297238] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.298043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.299831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.302015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.311018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.314017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.317000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.325016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.357067] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.367641] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.378017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.401016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.414130] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.421016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.428016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.443016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.451154] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.466015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.483016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.495269] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.499000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.508014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.530019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.538728] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.545016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.551017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.568020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.580000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.583415] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.586463] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.588492] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.591247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.597181] iommu: Default domain type: Passthrough [ 0.598459] SCSI subsystem initialized [ 0.599152] ACPI: bus type USB registered [ 0.601122] usbcore: registered new interface driver usbfs [ 0.603217] usbcore: registered new interface driver hub [ 0.605094] usbcore: registered new device driver usb [ 0.608201] pps_core: LinuxPPS API ver. 1 registered [ 0.610012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.614060] PTP clock support registered [ 0.616316] EDAC MC: Ver: 3.0.0 [ 0.618174] PCI: Using ACPI for IRQ routing [ 0.621096] NetLabel: Initializing [ 0.622011] NetLabel: domain hash size = 128 [ 0.624011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626242] NetLabel: unlabeled traffic allowed by default [ 0.629366] vgaarb: loaded [ 0.631625] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.633013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.641404] clocksource: Switched to clocksource kvm-clock [ 0.762475] VFS: Disk quotas dquot_6.6.0 [ 0.764110] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.766848] *** VALIDATE ramfs *** [ 0.768194] *** VALIDATE hugetlbfs *** [ 0.769934] pnp: PnP ACPI init [ 0.772519] pnp: PnP ACPI: found 6 devices [ 0.789084] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.792310] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.794139] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.795835] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.797883] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.800654] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.803778] NET: Registered protocol family 2 [ 0.806333] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.811698] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.815476] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820654] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.824592] TCP: Hash tables configured (established 65536 bind 65536) [ 0.827688] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.831026] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.834346] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.837485] NET: Registered protocol family 1 [ 0.840112] RPC: Registered named UNIX socket transport module. [ 0.842461] RPC: Registered udp transport module. [ 0.844195] RPC: Registered tcp transport module. [ 0.846299] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.849036] NET: Registered protocol family 44 [ 0.851012] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.853491] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.855740] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.858266] PCI: CLS 0 bytes, default 64 [ 0.860296] Unpacking initramfs... [ 3.229758] debug: unmapping init [mem 0xffff8de73cc54000-0xffff8de73ffbffff] [ 3.234676] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.240686] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.247953] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.146945] Initialise system trusted keyrings [ 4.148947] Key type blacklist registered [ 4.151032] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.161597] zbud: loaded [ 4.165049] *** VALIDATE nfs *** [ 4.166319] *** VALIDATE nfs4 *** [ 4.168223] pstore: using deflate compression [ 4.172434] Platform Keyring initialized [ 4.432900] NET: Registered protocol family 38 [ 4.438735] Key type asymmetric registered [ 4.442717] Asymmetric key parser 'x509' registered [ 4.447905] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.456368] io scheduler mq-deadline registered [ 4.459196] io scheduler kyber registered [ 4.468801] io scheduler bfq registered [ 4.472687] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.478943] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.487802] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.495970] ACPI: Power Button [PWRF] [ 4.509895] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.522540] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.557584] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.575744] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.616811] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.657442] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.691073] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.697988] Non-volatile memory driver v1.3 [ 4.700855] Linux agpgart interface v0.103 [ 4.747126] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.750464] vda: detected capacity change from 0 to 74698752 [ 4.783809] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.789730] vdb: detected capacity change from 0 to 1073741824 [ 4.821760] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.826147] vdc: detected capacity change from 0 to 2621440000 [ 4.855306] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.860241] vdd: detected capacity change from 0 to 2621440000 [ 4.882560] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.887287] vde: detected capacity change from 0 to 4294967296 [ 4.915622] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.919842] vdf: detected capacity change from 0 to 4294967296 [ 4.938767] libphy: Fixed MDIO Bus: probed [ 4.957263] usbcore: registered new interface driver usbserial_generic [ 4.962516] usbserial: USB Serial support registered for generic [ 4.964870] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.969543] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.971586] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.974293] mousedev: PS/2 mouse device common for all mice [ 4.977645] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.980418] rtc_cmos 00:05: RTC can wake from S4 [ 4.989432] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.991225] rtc_cmos 00:05: registered as rtc0 [ 5.011051] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.020702] intel_pstate: CPU model not supported [ 5.029080] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.039022] hid: raw HID events driver (C) Jiri Kosina [ 5.048130] usbcore: registered new interface driver usbhid [ 5.054134] usbhid: USB HID core driver [ 5.058550] drop_monitor: Initializing network drop monitor service [ 5.065071] Initializing XFRM netlink socket [ 5.070471] NET: Registered protocol family 10 [ 5.080040] Segment Routing with IPv6 [ 5.084079] NET: Registered protocol family 17 [ 5.091661] mpls_gso: MPLS GSO support [ 5.108646] RAS: Correctable Errors collector initialized. [ 5.115081] AVX version of gcm_enc/dec engaged. [ 5.119234] AES CTR mode by8 optimization enabled [ 5.400241] sched_clock: Marking stable (5400214825, 0)->(6896755190, -1496540365) [ 5.412198] registered taskstats version 1 [ 5.418188] Loading compiled-in X.509 certificates [ 5.422907] zswap: loaded using pool lzo/zbud [ 5.510432] Key type big_key registered [ 5.536469] Key type encrypted registered [ 5.538589] ima: No TPM chip found, activating TPM-bypass! [ 5.541436] ima: Allocated hash algorithm: sha1 [ 5.543614] ima: No architecture policies found [ 5.546111] evm: Initialising EVM extended attributes: [ 5.548660] evm: security.selinux [ 5.550177] evm: security.ima [ 5.551451] evm: security.capability [ 5.552912] evm: HMAC attrs: 0x1 [ 5.559585] rtc_cmos 00:05: setting system clock to 2026-08-08 06:14:58 UTC (1786169698) [ 5.570565] debug: unmapping init [mem 0xffffffff8fa03000-0xffffffff8fbfffff] [ 5.574958] debug: unmapping init [mem 0xffffffff8e782000-0xffffffff8ea58fff] [ 5.587167] Write protecting the kernel read-only data: 28672k [ 5.599381] debug: unmapping init [mem 0xffffffff8ce03000-0xffffffff8cffffff] [ 5.607032] debug: unmapping init [mem 0xffffffff8d714000-0xffffffff8d7fffff] [ 5.682676] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.705083] systemd[1]: Detected virtualization kvm. [ 5.708046] systemd[1]: Detected architecture x86-64. [ 5.715128] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.768957] systemd[1]: No hostname configured. [ 5.773649] systemd[1]: Set hostname to . [ 5.776061] random: systemd: uninitialized urandom read (16 bytes read) [ 5.779648] systemd[1]: Initializing machine ID from random generator. [ 6.060191] random: ln: uninitialized urandom read (6 bytes read) [ 6.493224] random: systemd: uninitialized urandom read (16 bytes read) [ 6.503249] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.527356] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 6.553989] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.481659] device-mapper: uevent: version 1.0.3 [ 8.485845] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 10.217756] virtio_net virtio0 ens2: renamed from eth0 [ 10.225227] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.597794] scsi host0: ata_piix [ 10.603863] scsi host1: ata_piix [ 10.607745] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.611647] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.073598] random: crng init done [ 15.078620] random: 7 urandom warning(s) missed due to ratelimiting [ 17.881798] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.273204] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.532789] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.359107] SELinux: Disabled at runtime. [ 24.485290] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.506542] systemd[1]: Detected virtualization kvm. [ 24.514888] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.802716] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.809312] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.825878] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.837616] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.844406] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.857683] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.872280] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ 26.630718] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.867933] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.153801] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.236347] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.603709] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.849189] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.465891] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 36.612858] NFS: Registering the id_resolver key type [ 36.619577] Key type id_resolver registered [ 36.625278] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg452-server login: [ 66.251050] hrtimer: interrupt took 10040315 ns [ 108.343828] libcfs: loading out-of-tree module taints kernel. [ 108.383540] Key type ._llcrypt registered [ 108.385593] Key type .llcrypt registered [ 108.469214] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 128.002786] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 130.587123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 130.610919] alg: No test for adler32 (adler32-zlib) [ 132.289433] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 133.285983] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 135.103159] Key type lgssc registered [ 137.227469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.700982] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.098270] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 215.518970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.562397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.899593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.933871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.070410] Lustre: lustre-MDT0000: new disk, initializing [ 217.159775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.177866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.127448] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.606205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 234.741640] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 234.773576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 234.777260] Lustre: Skipped 1 previous similar message [ 234.866366] Lustre: lustre-MDT0001: new disk, initializing [ 234.942675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 234.989864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 235.013190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 240.446405] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.065690] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.156417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.503702] Lustre: lustre-OST0000: new disk, initializing [ 256.515951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.539345] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.636060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.249045] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 266.797792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 266.823516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 266.947256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 279.298502] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 279.472204] Lustre: lustre-OST0001: new disk, initializing [ 279.479706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 279.498994] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 279.607120] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 286.041280] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.301466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.323615] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.416852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 298.901504] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 305.007233] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 312.129147] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.019732] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing yml_node [ 322.337236] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 324.836918] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 327.520809] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 329.512230] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 02:20:21 EDT 2026 [ 350.427143] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 352.659373] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 368.101386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.107498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 368.116931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.633133] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 369.639715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.669179] Lustre: Skipped 3 previous similar messages [ 374.370300] Lustre: server umount lustre-MDT0000 complete [ 378.122493] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170071 with bad export cookie 1018817477977137337 [ 378.123290] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.138647] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 379.871836] LustreError: 9537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 379.873073] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.873484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 379.902042] LustreError: 9537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 379.933732] Lustre: Skipped 3 previous similar messages [ 383.904734] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.904756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 383.941566] Lustre: Skipped 2 previous similar messages [ 384.552337] Lustre: server umount lustre-MDT0001 complete [ 389.242479] Lustre: server umount lustre-OST0000 complete [ 393.658938] Lustre: server umount lustre-OST0001 complete [ 401.582356] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 416.518366] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 454.076807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.448331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 454.495777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 454.570415] Lustre: lustre-MDT0000: new disk, initializing [ 454.677198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.715064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 459.679212] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 471.638937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.768475] Lustre: 15108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.824407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.827911] Lustre: Skipped 1 previous similar message [ 471.889766] Lustre: lustre-MDT0001: new disk, initializing [ 472.005671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 472.088687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 472.116781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 477.572639] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 489.562873] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.550340] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 507.065827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 507.410782] Lustre: lustre-OST0000: new disk, initializing [ 507.415711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 507.421262] Lustre: 17081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 507.573719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.520941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 509.565380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 509.875879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 515.671907] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 525.816864] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 530.298199] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 530.537644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 537.190585] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 537.435221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 540.648595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.651109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.690902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.691052] Lustre: Skipped 1 previous similar message [ 540.707248] LustreError: Skipped 1 previous similar message [ 540.740678] Lustre: Skipped 1 previous similar message [ 546.103726] Lustre: server umount lustre-OST0000 complete [ 550.883145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.884220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 550.892055] Lustre: Skipped 1 previous similar message [ 550.902606] Lustre: Skipped 3 previous similar messages [ 556.797586] Lustre: server umount lustre-MDT0000 complete [ 561.150118] LustreError: 15995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.179326] LustreError: 15995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 561.423790] LustreError: 15101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170254 with bad export cookie 1018817477977138870 [ 561.434628] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.443643] LustreError: 15101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 561.940104] Lustre: server umount lustre-MDT0001 complete [ 565.981841] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 02:24:17 (1786170257) [ 572.545270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.026463] LustreError: 19345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.110293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.971553] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 578.530435] LustreError: 19346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.239526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.603536] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 600.479878] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 607.785983] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.393206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 625.411465] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 634.016812] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 642.537201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 642.543662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.561893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 642.577441] Lustre: Skipped 1 previous similar message [ 642.587346] Lustre: Skipped 3 previous similar messages [ 648.224097] Lustre: server umount lustre-OST0000 complete [ 658.475728] Lustre: server umount lustre-MDT0000 complete [ 662.149905] LustreError: 20081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170355 with bad export cookie 1018817477977139752 [ 662.152469] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.165130] LustreError: 20081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 662.531409] Lustre: server umount lustre-MDT0001 complete [ 673.826717] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 676.300450] Key type lgssc unregistered [ 676.580776] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.589992] LNetError: 22881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.671596] LNet: Removed LNI 192.168.204.152@tcp [ 679.168484] Key type .llcrypt unregistered [ 679.172116] Key type ._llcrypt unregistered [ 706.779338] Key type ._llcrypt registered [ 706.781803] Key type .llcrypt registered [ 706.895982] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 708.177185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 708.522231] alg: No test for adler32 (adler32-zlib) [ 710.087632] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 710.567900] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 712.223256] Key type lgssc registered [ 713.789813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 722.340501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 722.378846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.992148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.232954] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 733.156181] LustreError: 24298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.167712] LustreError: 24298:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 738.155961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.286779] LustreError: 24299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 739.034506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.940685] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 751.545137] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 757.638362] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 764.182678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 764.560037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.703848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 770.192398] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 778.203063] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 793.575334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 793.590390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 793.595662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 793.618293] Lustre: Skipped 1 previous similar message [ 798.534805] Lustre: server umount lustre-OST0000 complete [ 803.810574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 803.813369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 803.829550] Lustre: Skipped 1 previous similar message [ 803.845277] Lustre: Skipped 2 previous similar messages [ 808.945098] LustreError: 25513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.984674] LustreError: 25513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 809.092854] Lustre: server umount lustre-MDT0000 complete [ 812.732690] LustreError: 24279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170505 with bad export cookie 7991283262722582335 [ 812.733952] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 812.757164] LustreError: 24279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 813.067194] Lustre: server umount lustre-MDT0001 complete [ 825.562620] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 828.520786] Key type lgssc unregistered [ 828.809446] LNet: 28276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.821954] LNetError: 28276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.864528] LNet: Removed LNI 192.168.204.152@tcp [ 830.714361] Key type .llcrypt unregistered [ 830.716118] Key type ._llcrypt unregistered [ 851.770975] Key type ._llcrypt registered [ 851.772637] Key type .llcrypt registered [ 851.900469] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 852.861359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.884936] alg: No test for adler32 (adler32-zlib) [ 854.129220] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 854.550387] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 856.375169] Key type lgssc registered [ 857.410687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 863.972771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 863.987595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.333335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 870.005215] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 875.493966] LustreError: 29669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 875.511171] LustreError: 29669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 878.711178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.999610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 884.565738] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 893.308792] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.884564] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 911.428849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 911.779774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.191176] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 925.329808] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 935.354506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 944.611189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 944.617727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.625918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 945.639230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.643861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 945.648608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.552878] Lustre: server umount lustre-OST0000 complete [ 955.875990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.878332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 955.878663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 955.878668] Lustre: Skipped 1 previous similar message [ 955.892568] Lustre: Skipped 1 previous similar message [ 960.996959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 961.009877] Lustre: Skipped 2 previous similar messages [ 966.111129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 966.112546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.130900] Lustre: Skipped 1 previous similar message [ 966.345608] Lustre: server umount lustre-MDT0000 complete [ 970.212108] LustreError: 30396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.236068] LustreError: 30396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 970.549000] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170663 with bad export cookie 18408725060617633531 [ 970.557155] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.566246] LustreError: 29649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 971.072850] Lustre: server umount lustre-MDT0001 complete [ 983.570528] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 986.708582] Key type lgssc unregistered [ 987.271514] LNet: 33572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.284877] LNetError: 33572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.308485] LNet: Removed LNI 192.168.204.152@tcp [ 988.585659] Key type .llcrypt unregistered [ 988.588770] Key type ._llcrypt unregistered [ 1015.299447] Key type ._llcrypt registered [ 1015.301235] Key type .llcrypt registered [ 1015.425704] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1017.456328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.503472] alg: No test for adler32 (adler32-zlib) [ 1019.047908] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1019.257913] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1021.023864] Key type lgssc registered [ 1022.370737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1030.036415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1030.059660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.388800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.634243] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1041.904242] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.949822] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1046.213497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.483490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1051.958940] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1059.562329] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.532678] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1069.652501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.999243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.774289] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1076.174540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1084.132893] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.576730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1092.586560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.620122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.683519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1096.684016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1096.688043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.563792] Lustre: server umount lustre-OST0000 complete [ 1102.817423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1102.825378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.847887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.853442] Lustre: Skipped 1 previous similar message [ 1106.921622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.931133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.332137] Lustre: server umount lustre-MDT0000 complete [ 1112.032987] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1112.058382] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1112.283326] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170805 with bad export cookie 16742464865122967884 [ 1112.289317] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.301015] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1112.651147] Lustre: server umount lustre-MDT0001 complete [ 1123.974612] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1126.788484] Key type lgssc unregistered [ 1127.118269] LNet: 38524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.133757] LNetError: 38524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.159546] LNet: Removed LNI 192.168.204.152@tcp [ 1128.109493] Key type .llcrypt unregistered [ 1128.114267] Key type ._llcrypt unregistered [ 1155.500199] Key type ._llcrypt registered [ 1155.505330] Key type .llcrypt registered [ 1155.674782] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1157.577628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.749523] alg: No test for adler32 (adler32-zlib) [ 1159.045304] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1159.260487] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1160.905870] Key type lgssc registered [ 1162.343242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1171.953938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1171.979434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.345599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.138889] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1183.733320] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.766431] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1186.828200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.125020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1191.749805] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1199.605802] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1204.941455] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1210.971837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1211.262753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.437268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1218.870480] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1226.777641] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1232.868208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.882339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.899535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.894469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.898185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1233.900451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.743385] Lustre: server umount lustre-OST0000 complete [ 1263.587853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.591117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.599670] Lustre: Skipped 1 previous similar message [ 1263.612617] Lustre: Skipped 3 previous similar messages [ 1265.656868] Lustre: server umount lustre-MDT0000 complete [ 1268.715587] LustreError: 40667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.748500] LustreError: 40667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1269.796863] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170962 with bad export cookie 2070792892870949413 [ 1269.802598] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.807280] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1276.214437] Lustre: server umount lustre-MDT0001 complete [ 1290.673854] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1294.747927] Key type lgssc unregistered [ 1295.109937] LNet: 43653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.134975] LNetError: 43653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.167085] LNet: Removed LNI 192.168.204.152@tcp [ 1296.347361] Key type .llcrypt unregistered [ 1296.352421] Key type ._llcrypt unregistered [ 1326.024921] Key type ._llcrypt registered [ 1326.033512] Key type .llcrypt registered [ 1326.203620] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1327.671022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.750592] alg: No test for adler32 (adler32-zlib) [ 1329.092459] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1329.314490] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1331.031240] Key type lgssc registered [ 1332.853356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1341.393569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1341.419612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.735408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.231285] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1352.178041] LustreError: 45068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.210730] LustreError: 45068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1355.992747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.226247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1361.020704] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1369.158641] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1374.449081] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1381.641748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1381.796494] LustreError: 46892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.012551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.204513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1388.343640] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1395.517677] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1402.338752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1402.342698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.350681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.853078] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.865693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.874143] Lustre: Skipped 1 previous similar message [ 1404.570634] Lustre: server umount lustre-MDT0000 complete [ 1407.973796] LustreError: 45063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.019431] LustreError: 45063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1408.741668] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171101 with bad export cookie 1173139351768742168 [ 1408.753047] LustreError: 45049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1408.753346] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1413.091557] LustreError: 45064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1413.093907] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.113911] LustreError: 45064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1413.132384] Lustre: Skipped 1 previous similar message [ 1413.137420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1413.150851] Lustre: Skipped 1 previous similar message [ 1418.207917] LustreError: 45065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1418.211722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1418.231123] LustreError: 45065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1418.251874] Lustre: Skipped 1 previous similar message [ 1423.327336] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1423.697286] Lustre: server umount lustre-MDT0001 complete [ 1439.280988] Lustre: server umount lustre-OST0000 complete [ 1458.161648] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1461.274282] Key type lgssc unregistered [ 1461.688470] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.698442] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.722469] LNet: Removed LNI 192.168.204.152@tcp [ 1462.643541] Key type .llcrypt unregistered [ 1462.646742] Key type ._llcrypt unregistered [ 1487.982487] Key type ._llcrypt registered [ 1487.985478] Key type .llcrypt registered [ 1488.111967] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1489.239745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.388203] alg: No test for adler32 (adler32-zlib) [ 1490.501986] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1490.755284] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1492.415244] Key type lgssc registered [ 1494.075939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1501.729407] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1501.746856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.343418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1522.463589] LustreError: 50323:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 1549.087215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1555.825378] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1563.151964] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1729.207981] Lustre: server umount lustre-OST0000 complete [ 1750.854930] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1754.277638] Key type lgssc unregistered [ 1754.572819] LNet: 52001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.581857] LNetError: 52001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.601524] LNet: Removed LNI 192.168.204.152@tcp [ 1755.617677] Key type .llcrypt unregistered [ 1755.620376] Key type ._llcrypt unregistered [ 1784.910411] Key type ._llcrypt registered [ 1784.919719] Key type .llcrypt registered [ 1785.135301] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1786.705580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.892519] alg: No test for adler32 (adler32-zlib) [ 1788.125842] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1788.488469] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1790.280644] Key type lgssc registered [ 1792.086020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1801.240214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1801.279436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.004620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.932181] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1812.455931] LustreError: 53423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1812.478269] LustreError: 53423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1817.575985] LustreError: 53422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.323639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.639198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.515284] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1832.710457] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1839.653786] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1848.336096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.641269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.103627] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 1863.830730] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1869.281688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1869.285646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.296690] LustreError: Skipped 1 previous similar message [ 1869.342249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1872.424195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1872.845906] Lustre: server umount lustre-OST0000 complete [ 1877.472027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.495952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.510668] Lustre: Skipped 2 previous similar messages [ 1882.592755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.598872] Lustre: Skipped 1 previous similar message [ 1883.485203] Lustre: server umount lustre-MDT0000 complete [ 1887.715857] LustreError: 53422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.731109] LustreError: 53422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1888.149226] LustreError: 53401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171581 with bad export cookie 7030020783310412948 [ 1888.152109] LustreError: 54152:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 1888.157614] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.165594] LustreError: 53401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1888.203687] LustreError: 54152:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 1888.462174] Lustre: server umount lustre-MDT0001 complete [ 1904.333847] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 1908.676350] Key type lgssc unregistered [ 1909.025602] LNet: 56956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.045011] LNetError: 56956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.072935] LNet: Removed LNI 192.168.204.152@tcp [ 1910.404781] Key type .llcrypt unregistered [ 1910.411172] Key type ._llcrypt unregistered [ 1941.831707] Key type ._llcrypt registered [ 1941.834134] Key type .llcrypt registered [ 1941.969780] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 1943.512290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1943.646486] alg: No test for adler32 (adler32-zlib) [ 1944.864391] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 1945.237445] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 1946.967197] Key type lgssc registered [ 1948.219156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1957.691171] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1957.711711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1973.216155] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1978.335483] LustreError: 58345:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 2005.023465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.998775] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2020.891637] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2027.516035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.058854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.145310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2032.819569] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2033.138856] LustreError: 58371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.163656] LustreError: 58371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2038.244367] LustreError: 58370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.274987] LustreError: 58370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2039.289622] LustreError: 58370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2042.431992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.781550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.831508] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2058.506942] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.471305] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2068.454685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2068.466704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.478163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2068.964711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2068.969667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.985073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.538083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2073.675864] Lustre: server umount lustre-OST0000 complete [ 2099.170108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.174625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.181670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.185344] Lustre: Skipped 1 previous similar message [ 2101.220042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.204234] Lustre: server umount lustre-MDT0000 complete [ 2106.340166] LustreError: 59386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.360699] LustreError: 59386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2106.821961] LustreError: 58352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171799 with bad export cookie 7531491426552219884 [ 2106.836522] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.841071] LustreError: 58352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2121.183249] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2121.547095] Lustre: server umount lustre-MDT0001 complete [ 2135.146843] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2138.260304] Key type lgssc unregistered [ 2138.475826] LNet: 62095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.488991] LNetError: 62095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.509534] LNet: Removed LNI 192.168.204.152@tcp [ 2139.391626] Key type .llcrypt unregistered [ 2139.393726] Key type ._llcrypt unregistered [ 2167.645709] Key type ._llcrypt registered [ 2167.653291] Key type .llcrypt registered [ 2167.832700] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2170.994405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.036578] alg: No test for adler32 (adler32-zlib) [ 2172.552152] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2173.002290] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2174.752963] Key type lgssc registered [ 2176.660352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2187.123914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2187.171893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.685335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.791377] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2199.015069] LustreError: 63509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.046539] LustreError: 63509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2204.157661] LustreError: 63510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.088194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.572269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.869939] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2222.217230] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2229.382713] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2238.501182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2238.789622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.868466] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2254.060754] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2258.252490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2272.741395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2272.754339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.777266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2272.956639] Lustre: server umount lustre-OST0000 complete [ 2278.370220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.380646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.404081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.001393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.005800] Lustre: Skipped 2 previous similar messages [ 2288.103122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.111516] Lustre: Skipped 1 previous similar message [ 2291.167265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2291.318585] Lustre: server umount lustre-MDT0000 complete [ 2293.223670] LustreError: 63510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.234452] LustreError: 63510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2294.785927] LustreError: 63491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171987 with bad export cookie 3557964729358245999 [ 2294.822553] LustreError: 63491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2294.824824] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.158629] Lustre: server umount lustre-MDT0001 complete [ 2309.269813] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2312.431313] Key type lgssc unregistered [ 2312.938539] LNet: 67074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.955893] LNetError: 67074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.972416] LNet: Removed LNI 192.168.204.152@tcp [ 2313.785638] Key type .llcrypt unregistered [ 2313.790907] Key type ._llcrypt unregistered [ 2343.023919] Key type ._llcrypt registered [ 2343.026968] Key type .llcrypt registered [ 2343.149160] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2344.401706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.462270] alg: No test for adler32 (adler32-zlib) [ 2345.675222] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2346.007109] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2347.663152] Key type lgssc registered [ 2349.123033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2360.853620] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:54:12 (1786172052) [ 2368.133115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.178375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.802759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.742237] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2379.239638] LustreError: 68779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.255061] LustreError: 68779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2384.369068] LustreError: 68780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.349256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.840338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2392.221142] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2400.402072] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2406.686360] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.918932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.408807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.510583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2422.167529] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2430.590890] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.931448] Lustre: Failing over lustre-MDT0000 [ 2437.109738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.114547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.141718] Lustre: Skipped 2 previous similar messages [ 2437.147263] Lustre: Skipped 1 previous similar message [ 2437.284881] Lustre: server umount lustre-MDT0000 complete [ 2437.671623] LustreError: 68775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.215031] LustreError: 68774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.242971] LustreError: 68774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2447.337371] LustreError: 70075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.387147] LustreError: 70075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2448.072464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.207697] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.332577] Lustre: *** cfs_fail_loc=135, val=0*** [ 2448.470803] LustreError: 71866:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2448.477669] LustreError: 71866:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.152@tcp: cfg command failed: rc = -2 [ 2448.499813] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2448.499813] [ 2448.516418] LustreError: MGC192.168.204.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2448.554071] LustreError: 71854:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2448.572119] LustreError: 71854:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2448.592953] LustreError: 71854:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2448.617454] LustreError: 71854:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2448.665557] Lustre: server umount lustre-MDT0000 complete [ 2448.669355] LustreError: 71854:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2456.864106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.072245] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.106971] LustreError: 68770:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2457.573574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.695730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2459.182104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2462.711074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2462.765832] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2462.838995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2463.317329] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2472.929816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.957980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2472.963553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2472.994124] Lustre: Skipped 2 previous similar messages [ 2478.065269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.658837] Lustre: server umount lustre-OST0000 complete [ 2483.173950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.177779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2483.180933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.180945] Lustre: Skipped 1 previous similar message [ 2483.206529] Lustre: Skipped 2 previous similar messages [ 2488.291148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2488.302299] Lustre: Skipped 2 previous similar messages [ 2489.367858] Lustre: server umount lustre-MDT0000 complete [ 2493.410669] LustreError: 68776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.459446] LustreError: 68776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2493.725689] LustreError: 68760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172186 with bad export cookie 1663427486235792860 [ 2493.736587] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.747852] LustreError: 68760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2494.129525] Lustre: server umount lustre-MDT0001 complete [ 2507.865567] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2511.952660] Key type lgssc unregistered [ 2512.267254] LNet: 73651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2512.275710] LNetError: 73651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.319751] LNet: Removed LNI 192.168.204.152@tcp [ 2514.243983] Key type .llcrypt unregistered [ 2514.246671] Key type ._llcrypt unregistered [ 2538.977555] Key type ._llcrypt registered [ 2538.982621] Key type .llcrypt registered [ 2539.173862] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2540.533136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2540.841102] alg: No test for adler32 (adler32-zlib) [ 2541.990327] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2542.213938] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2543.895308] Key type lgssc registered [ 2545.334375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.264775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.305266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.673184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.577864] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2565.095349] LustreError: 75063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.126238] LustreError: 75063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2569.302748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.618992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2574.052140] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2582.249913] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.667595] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.295738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.590617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.777331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2600.885567] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2609.312183] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2618.536873] Lustre: Failing over lustre-MDT0000 [ 2618.857231] Lustre: server umount lustre-MDT0000 complete [ 2620.388462] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.394102] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.421570] Lustre: Skipped 1 previous similar message [ 2620.473258] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2621.477730] LustreError: 75059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.515802] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.561385] LustreError: 75060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2627.979235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.051142] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.153532] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2628.262151] LustreError: 78152:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2628.267317] LustreError: 78152:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.152@tcp: cfg command failed: rc = -9 [ 2628.274225] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2628.274225] [ 2628.288231] LustreError: MGC192.168.204.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2628.308589] LustreError: 78141:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2628.317604] LustreError: 78141:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2628.327686] LustreError: 78141:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2628.335930] LustreError: 78141:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2628.367493] Lustre: server umount lustre-MDT0000 complete [ 2628.374540] LustreError: 78141:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2630.638907] LustreError: 75058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.686471] LustreError: 75058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2634.968299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.082404] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.317960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.379433] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2636.812655] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2640.377906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2640.412872] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2640.451855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2640.917382] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2650.595686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.613081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2650.622493] Lustre: Skipped 2 previous similar messages [ 2650.638860] Lustre: Skipped 1 previous similar message [ 2655.702484] Lustre: server umount lustre-OST0000 complete [ 2660.833148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.834209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2660.835551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.841745] Lustre: Skipped 1 previous similar message [ 2665.964061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.971957] Lustre: Skipped 2 previous similar messages [ 2666.298571] Lustre: server umount lustre-MDT0000 complete [ 2670.598824] LustreError: 75043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172363 with bad export cookie 7690302668664969643 [ 2670.616848] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.627306] LustreError: 75043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2670.966710] Lustre: server umount lustre-MDT0001 complete [ 2684.670395] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2688.256932] Key type lgssc unregistered [ 2688.632865] LNet: 79935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2688.645265] LNetError: 79935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2688.661565] LNet: Removed LNI 192.168.204.152@tcp [ 2689.447116] Key type .llcrypt unregistered [ 2689.450756] Key type ._llcrypt unregistered [ 2702.917884] Key type ._llcrypt registered [ 2702.920405] Key type .llcrypt registered [ 2703.073644] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2703.533567] Key type .llcrypt unregistered [ 2703.538414] Key type ._llcrypt unregistered [ 2729.761376] Key type ._llcrypt registered [ 2729.766381] Key type .llcrypt registered [ 2729.889803] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2730.894001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2731.016135] alg: No test for adler32 (adler32-zlib) [ 2732.122505] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2732.450967] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2734.143181] Key type lgssc registered [ 2735.648180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.497865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2743.520855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.918551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.483528] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2755.047127] LustreError: 81855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.106433] LustreError: 81855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2759.833195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.085273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2764.626862] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2773.915191] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2780.150891] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2788.334906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.637571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.727564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2795.843470] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2803.192395] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2814.436479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.456495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.463093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.483030] Lustre: Skipped 1 previous similar message [ 2814.494863] Lustre: Skipped 1 previous similar message [ 2818.445136] Lustre: server umount lustre-OST0000 complete [ 2824.678860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.684785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.689914] Lustre: Skipped 1 previous similar message [ 2824.709513] Lustre: Skipped 1 previous similar message [ 2829.666109] Lustre: server umount lustre-MDT0000 complete [ 2829.796057] LustreError: 84359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.844718] LustreError: 84359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2834.046968] LustreError: 81834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172526 with bad export cookie 1547374833075056283 [ 2834.058802] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.071446] LustreError: 81834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2834.548858] Lustre: server umount lustre-MDT0001 complete [ 2848.107662] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2850.941506] Key type lgssc unregistered [ 2851.246875] LNet: 85405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2851.253847] LNetError: 85405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2851.267114] LNet: Removed LNI 192.168.204.152@tcp [ 2852.032838] Key type .llcrypt unregistered [ 2852.038150] Key type ._llcrypt unregistered [ 2880.236087] Key type ._llcrypt registered [ 2880.238231] Key type .llcrypt registered [ 2880.391810] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 2882.185789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2882.396053] alg: No test for adler32 (adler32-zlib) [ 2883.653900] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 2883.971670] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 2885.695194] Key type lgssc registered [ 2886.875880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2896.271517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2896.297778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.770141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.774572] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2907.120834] LustreError: 86820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.148344] LustreError: 86820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2911.132397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.415755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2915.867352] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2923.487122] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2929.811607] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2936.577932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.870233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2943.034807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2944.723642] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 2953.958461] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2963.428103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2963.440848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.459339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.506683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2966.508730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2966.541307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.358627] Lustre: server umount lustre-OST0000 complete [ 2976.738398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.745714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.757988] Lustre: Skipped 1 previous similar message [ 2976.774992] Lustre: Skipped 2 previous similar messages [ 2980.536054] Lustre: server umount lustre-MDT0000 complete [ 2981.879048] LustreError: 86820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.902036] LustreError: 86820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2984.229384] LustreError: 86801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172677 with bad export cookie 3097171321254807215 [ 2984.234507] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.239335] LustreError: 86801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2984.468992] Lustre: server umount lustre-MDT0001 complete [ 2996.759661] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 2999.545412] Key type lgssc unregistered [ 2999.915335] LNet: 90357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2999.926781] LNetError: 90357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2999.944913] LNet: Removed LNI 192.168.204.152@tcp [ 3001.050449] Key type .llcrypt unregistered [ 3001.054585] Key type ._llcrypt unregistered [ 3027.222718] Key type ._llcrypt registered [ 3027.237452] Key type .llcrypt registered [ 3027.405096] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3029.820332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3029.861345] alg: No test for adler32 (adler32-zlib) [ 3030.984435] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 3031.302873] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3033.008306] Key type lgssc registered [ 3034.354117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.211397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3043.228808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.574245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.931374] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3055.077376] LustreError: 91773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.096419] LustreError: 91773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3060.203623] LustreError: 91774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.818581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.411223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.111857] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3076.159666] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3083.122815] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3091.986048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.605487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.636559] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3108.085831] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3115.142304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3130.878343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.878566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.915747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.926290] Lustre: Skipped 1 previous similar message [ 3131.714852] Lustre: server umount lustre-OST0000 complete [ 3135.972171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.983389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3136.004713] Lustre: Skipped 2 previous similar messages [ 3136.032028] Lustre: Skipped 1 previous similar message [ 3137.876816] Lustre: server umount lustre-MDT0000 complete [ 3141.090915] LustreError: 91769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.134191] LustreError: 91769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3142.022877] LustreError: 94274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172834 with bad export cookie 1267016744480136361 [ 3142.034603] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.051758] LustreError: 94274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3142.341869] Lustre: server umount lustre-MDT0001 complete [ 3156.134308] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3160.241253] Key type lgssc unregistered [ 3160.561856] LNet: 95330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3160.570113] LNetError: 95330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3160.583820] LNet: Removed LNI 192.168.204.152@tcp [ 3161.299593] Key type .llcrypt unregistered [ 3161.306375] Key type ._llcrypt unregistered [ 3192.868305] Key type ._llcrypt registered [ 3192.869878] Key type .llcrypt registered [ 3193.152351] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3195.706632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3195.772087] alg: No test for adler32 (adler32-zlib) [ 3197.178913] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 3197.532318] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3199.215167] Key type lgssc registered [ 3201.522545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.927792] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3211.979717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.680957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3232.799314] LustreError: 96721:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 3259.423432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.251572] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3275.189978] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3281.719508] Lustre: server umount lustre-OST0000 complete [ 3291.583847] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 03:09:43 (1786172983) [ 3299.421856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.789588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3304.931295] LustreError: 98363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.979542] LustreError: 98363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3305.158538] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3310.063082] LustreError: 98364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.762666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.057602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3319.749995] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3327.947903] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3334.675925] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3342.335433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.573604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.652698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3349.496639] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3357.403124] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3375.598618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.602414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.607731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3375.626033] Lustre: Skipped 1 previous similar message [ 3375.661804] LustreError: Skipped 1 previous similar message [ 3377.019823] Lustre: server umount lustre-OST0000 complete [ 3381.731218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3381.746667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.781480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.785639] Lustre: Skipped 1 previous similar message [ 3385.829959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.831169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.030775] Lustre: server umount lustre-MDT0000 complete [ 3390.947476] LustreError: 99835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.972161] LustreError: 99835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3392.513786] LustreError: 98344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173085 with bad export cookie 3319824738855999423 [ 3392.519828] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.522029] LustreError: 98344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3392.879281] Lustre: server umount lustre-MDT0001 complete [ 3405.993349] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3409.647504] Key type lgssc unregistered [ 3409.930857] LNet: 102072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.940534] LNetError: 102072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3410.982910] LNet: Removed LNI 192.168.204.152@tcp [ 3412.211597] Key type .llcrypt unregistered [ 3412.218310] Key type ._llcrypt unregistered [ 3462.646367] Key type ._llcrypt registered [ 3462.650675] Key type .llcrypt registered [ 3462.836209] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3464.607120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3464.684636] alg: No test for adler32 (adler32-zlib) [ 3466.053776] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 3466.538306] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3468.239536] Key type lgssc registered [ 3469.320156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3478.772826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3478.807893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.621934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.587814] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3490.785791] LustreError: 104262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.815424] LustreError: 104262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3495.912991] LustreError: 104261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.064809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.511654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3505.886894] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3514.995496] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3521.249808] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3529.281622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3529.577360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.777567] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3545.739817] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3551.115952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3560.419115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3560.422912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.428980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3561.442056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3561.442437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3561.451445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3566.317963] Lustre: server umount lustre-OST0000 complete [ 3571.172098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3571.185691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.208593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.220759] Lustre: Skipped 1 previous similar message [ 3576.364436] Lustre: server umount lustre-MDT0000 complete [ 3576.807655] LustreError: 104262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.834098] LustreError: 104262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3580.507097] LustreError: 104240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173273 with bad export cookie 8536756806218833877 [ 3580.512763] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.540210] LustreError: 104240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3580.825922] Lustre: server umount lustre-MDT0001 complete [ 3595.784820] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 3599.413942] Key type lgssc unregistered [ 3599.732330] LNet: 107825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3599.743981] LNetError: 107825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3599.764554] LNet: Removed LNI 192.168.204.152@tcp [ 3600.950590] Key type .llcrypt unregistered [ 3600.952894] Key type ._llcrypt unregistered [ 3623.681753] Key type ._llcrypt registered [ 3623.687871] Key type .llcrypt registered [ 3623.936138] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3626.050991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3626.440216] alg: No test for adler32 (adler32-zlib) [ 3627.490916] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 3627.726192] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 3629.367112] Key type lgssc registered [ 3631.119445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3641.254768] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3641.306013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.864390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3661.983335] LustreError: 109070:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 3688.671572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3696.011441] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3704.470744] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3713.843883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.117672] LustreError: MGC192.168.204.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3714.129484] LustreError: 110094:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3714.136156] LustreError: 110094:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3714.146827] LustreError: 110094:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3714.237228] Lustre: server umount lustre-MDT0000 complete [ 3714.241931] LustreError: 110094:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3732.959500] Lustre: 108407:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786173409/real 1786173409] req@0000000061b16e9b x1872938475195776/t0(0) o400->MGC192.168.204.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786173425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3732.979826] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.999430] LustreError: 109082:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3753.595223] Lustre: server umount lustre-OST0000 complete [ 3762.859394] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 3775.226503] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 3818.882458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.300959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3819.340985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3819.424687] Lustre: lustre-MDT0000: new disk, initializing [ 3819.497457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.524750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3826.081775] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3839.953272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3840.231961] Lustre: 112890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3840.352723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3840.363044] Lustre: Skipped 1 previous similar message [ 3840.475708] Lustre: lustre-MDT0001: new disk, initializing [ 3840.647610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.726465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3840.749657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3848.561585] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3860.531511] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3868.147588] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3876.089590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3876.501881] Lustre: lustre-OST0000: new disk, initializing [ 3876.509559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3876.520170] Lustre: 114861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3876.695092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.403775] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 3885.146521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3885.158192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3885.339454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3896.918527] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3903.199334] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3903.548061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3908.944717] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3909.199258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3916.261188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3916.267906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.274216] LustreError: Skipped 1 previous similar message [ 3916.285899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3916.296185] Lustre: Skipped 1 previous similar message [ 3916.322661] Lustre: Skipped 1 previous similar message [ 3917.806404] Lustre: server umount lustre-OST0000 complete [ 3921.380263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.398439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3921.398700] Lustre: Skipped 1 previous similar message [ 3921.412295] Lustre: Skipped 1 previous similar message [ 3926.496923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3926.508314] Lustre: Skipped 1 previous similar message [ 3927.472123] Lustre: server umount lustre-MDT0000 complete [ 3931.917492] LustreError: 112882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173624 with bad export cookie 9180784428251151098 [ 3931.919917] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3931.929367] LustreError: 112882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3932.189534] Lustre: server umount lustre-MDT0001 complete [ 3944.689238] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 03:20:36 (1786173636) [ 3949.517493] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3971.667603] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 3982.617830] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4023.564717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.747964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4023.770551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4023.858906] Lustre: lustre-MDT0000: new disk, initializing [ 4023.928235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.947885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4028.902721] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4039.870438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.969584] Lustre: 119765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4040.008847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4040.011802] Lustre: Skipped 1 previous similar message [ 4040.067222] Lustre: lustre-MDT0001: new disk, initializing [ 4040.139316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4040.173152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4040.184356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4045.766663] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4055.884415] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4063.313873] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4070.954449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.217252] Lustre: lustre-OST0000: new disk, initializing [ 4071.223251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4071.229963] Lustre: 121731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4071.305741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.272429] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4080.775757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4080.796118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4080.980658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4092.135662] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4097.173878] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4097.469401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4103.376429] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4103.570375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4106.724630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.727442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4106.737481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.747045] Lustre: Skipped 1 previous similar message [ 4111.842036] LustreError: 121715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.862850] LustreError: 121715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4111.907780] Lustre: server umount lustre-OST0000 complete [ 4116.965800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.979327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4116.993958] Lustre: Skipped 1 previous similar message [ 4116.995091] Lustre: Skipped 1 previous similar message [ 4127.202854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4127.212665] Lustre: Skipped 4 previous similar messages [ 4130.271579] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4130.417229] Lustre: server umount lustre-MDT0000 complete [ 4132.339231] LustreError: 119775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.371661] LustreError: 119775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4134.357856] LustreError: 120559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173827 with bad export cookie 9180784428251151980 [ 4134.363785] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.382622] LustreError: 120559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4134.753293] Lustre: server umount lustre-MDT0001 complete [ 4144.756499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4145.160842] LustreError: 123950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4145.242804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.244428] LustreError: 123951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.776343] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4155.369603] LustreError: 123950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.653380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.271691] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4174.765972] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.946717] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4187.886755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4194.968594] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4204.227818] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.750992] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4216.675820] Lustre: server umount lustre-OST0000 complete [ 4220.898094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.912462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4220.919558] Lustre: Skipped 1 previous similar message [ 4230.236796] Lustre: server umount lustre-MDT0000 complete [ 4231.139665] LustreError: 123950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.156139] LustreError: 123950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4234.194826] LustreError: 123930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173927 with bad export cookie 9180784428251152862 [ 4234.205414] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4234.206827] LustreError: 123930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4244.676227] Lustre: server umount lustre-MDT0001 complete [ 4257.660706] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 4261.275532] Key type lgssc unregistered [ 4261.567990] LNet: 127587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4261.578179] LNetError: 127587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4261.602434] LNet: Removed LNI 192.168.204.152@tcp [ 4262.722576] Key type .llcrypt unregistered [ 4262.724869] Key type ._llcrypt unregistered [ 4284.510533] Key type ._llcrypt registered [ 4284.519758] Key type .llcrypt registered [ 4284.707800] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 4300.667445] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 4302.101165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4302.546711] alg: No test for adler32 (adler32-zlib) [ 4303.669350] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 4303.963272] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 4305.767247] Key type lgssc registered [ 4306.937903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4345.039284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4345.085503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.438783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4346.474096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4346.552531] Lustre: lustre-MDT0000: new disk, initializing [ 4346.616946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.641675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4351.774783] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4363.232115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.377359] Lustre: 130742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4363.434586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4363.440511] Lustre: Skipped 1 previous similar message [ 4363.505765] Lustre: lustre-MDT0001: new disk, initializing [ 4363.611158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4363.647779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4363.666059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4368.874230] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4378.922745] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4384.815846] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4392.318654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4392.547867] Lustre: lustre-OST0000: new disk, initializing [ 4392.551919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4392.561787] Lustre: 132711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4392.645448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.673030] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4400.185847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4400.199032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4400.425380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4410.100911] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4414.975700] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4415.347432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4421.256360] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4421.490922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4426.215041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4426.215200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.220679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4426.228618] LustreError: Skipped 1 previous similar message [ 4426.272760] Lustre: Skipped 1 previous similar message [ 4429.579751] Lustre: server umount lustre-OST0000 complete [ 4435.427175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4435.437735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.452253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.460927] Lustre: Skipped 1 previous similar message [ 4436.456240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.472969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4436.477424] Lustre: Skipped 1 previous similar message [ 4440.593558] Lustre: server umount lustre-MDT0000 complete [ 4444.396685] LustreError: 131538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174137 with bad export cookie 16270894295994206314 [ 4444.416527] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4444.424658] LustreError: 131538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4444.764784] Lustre: server umount lustre-MDT0001 complete [ 4454.291681] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 03:29:06 (1786174146) [ 4461.671933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.984181] LustreError: 135124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.006413] LustreError: 135124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4462.064644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.333964] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4467.171155] LustreError: 135125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.190015] LustreError: 135125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4471.265785] LustreError: 135124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.795852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.287795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4481.815919] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4490.815314] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4498.265325] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4501.988151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.993495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4502.005602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.022206] Lustre: Skipped 1 previous similar message [ 4507.357607] Lustre: server umount lustre-MDT0000 complete [ 4508.138868] LustreError: 135124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.167019] LustreError: 135124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4510.994615] LustreError: 135105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174203 with bad export cookie 16270894295994207196 [ 4510.999450] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.007489] LustreError: 135105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4511.255978] Lustre: server umount lustre-MDT0001 complete [ 4520.806716] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 03:30:12 (1786174212) [ 4528.370860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4544.031853] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4549.151645] LustreError: 137539:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 4574.881124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4582.709284] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4590.375830] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.759476] Lustre: server umount lustre-OST0000 complete [ 4603.357332] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 03:31:34 (1786174294) [ 4610.403837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.739657] LustreError: 138986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.826134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.573806] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4620.277921] LustreError: 138987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.303139] LustreError: 138987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4626.334105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.260591] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4640.273463] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4647.168819] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.286668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4654.575874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.581983] Lustre: Skipped 1 previous similar message [ 4661.069831] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4669.148849] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4683.233656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4683.248208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.266458] Lustre: Skipped 1 previous similar message [ 4683.275828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4683.286774] Lustre: Skipped 2 previous similar messages [ 4688.733499] Lustre: server umount lustre-MDT0000 complete [ 4692.458672] LustreError: 138986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4692.487135] LustreError: 138986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4693.227626] LustreError: 140834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174386 with bad export cookie 16270894295994207854 [ 4693.246438] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.249984] LustreError: 140834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4693.646883] Lustre: server umount lustre-MDT0001 complete [ 4707.537893] Lustre: server umount lustre-OST0000 complete [ 4717.124054] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 03:33:28 (1786174408) [ 4723.748371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.093355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.970758] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4729.315542] LustreError: 142668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.329545] LustreError: 142668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4738.150655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.136744] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4751.244502] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4758.894790] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4766.993653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.381115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4774.256477] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4783.611986] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4790.210990] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4790.371789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4795.000294] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4795.321255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4799.457565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4799.462414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.467311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4799.467320] Lustre: Skipped 3 previous similar messages [ 4799.483929] LustreError: Skipped 1 previous similar message [ 4799.529155] Lustre: Skipped 3 previous similar messages [ 4803.656521] Lustre: server umount lustre-OST0000 complete [ 4814.366090] Lustre: server umount lustre-MDT0000 complete [ 4814.819894] LustreError: 143671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4814.839933] LustreError: 143671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4819.039596] LustreError: 142648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174511 with bad export cookie 16270894295994208932 [ 4819.056714] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.060661] LustreError: 142648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4819.483588] Lustre: server umount lustre-MDT0001 complete [ 4832.395451] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 03:35:23 (1786174523) [ 4841.431774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4857.055739] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4862.175635] LustreError: 146828:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 4887.839317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.843801] Lustre: Skipped 2 previous similar messages [ 4894.487092] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4903.332939] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4911.909085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.666779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4917.626451] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4926.781793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.980764] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 4940.093033] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4946.003896] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4952.161053] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4952.502285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4958.788563] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4959.219531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4963.300347] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4963.310469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4963.335565] Lustre: Skipped 3 previous similar messages [ 4963.351931] Lustre: Skipped 5 previous similar messages [ 4967.605908] Lustre: server umount lustre-MDT0000 complete [ 4968.423900] LustreError: 148878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.447765] LustreError: 148878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4971.440416] LustreError: 146836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174664 with bad export cookie 16270894295994209954 [ 4971.448970] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.454645] LustreError: 146836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4971.775376] Lustre: server umount lustre-MDT0001 complete [ 4986.109697] Lustre: server umount lustre-OST0000 complete [ 4995.262540] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 03:38:07 (1786174687) [ 5001.817772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5017.377267] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5022.496427] LustreError: 151025:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 5056.559910] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5065.475993] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5073.605783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.008088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5079.955105] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5089.481277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.278743] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5102.668063] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5109.900601] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5117.368145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5117.550368] Lustre: lustre-OST0001: new disk, initializing [ 5117.553919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5117.559964] Lustre: 153898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5124.858885] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5126.293288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5126.314973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5126.524656] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5136.610698] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5141.950324] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5142.208688] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5147.076248] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5147.408435] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5150.688852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.696085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.707325] Lustre: Skipped 1 previous similar message [ 5150.714286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5155.932453] Lustre: server umount lustre-OST0000 complete [ 5174.239531] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5174.491672] Lustre: server umount lustre-OST0001 complete [ 5181.922014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5181.931130] LustreError: Skipped 3 previous similar messages [ 5185.701950] Lustre: server umount lustre-MDT0000 complete [ 5189.834907] LustreError: 153897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174882 with bad export cookie 16270894295994210906 [ 5189.837460] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.856849] LustreError: 153897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5190.189838] Lustre: server umount lustre-MDT0001 complete [ 5216.573039] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 03:41:48 (1786174908) [ 5218.678966] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5220.502070] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5222.022196] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 03:41:54 (1786174914) [ 5229.939381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.212462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5230.289352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5230.583435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.593287] Lustre: Skipped 6 previous similar messages [ 5235.914800] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5244.753609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.838826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5244.863234] Lustre: 157458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5244.922129] LustreError: 157471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.946194] LustreError: 157471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 5250.134739] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5258.679473] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5265.063144] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5270.847198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5271.055543] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5277.605927] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5280.357940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5285.189685] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5289.855486] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5290.141794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5294.106111] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5294.328315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5300.714238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.714491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5300.729437] Lustre: Skipped 5 previous similar messages [ 5300.734592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5300.758810] Lustre: Skipped 13 previous similar messages [ 5302.585422] Lustre: server umount lustre-OST0000 complete [ 5326.118495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.361081] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5340.856306] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.876406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.882402] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5340.906202] Lustre: Skipped 1 previous similar message [ 5342.317155] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5348.384385] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5348.786305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5354.545796] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5354.984137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5361.536550] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c19d847d000.ost_server_uuid 50 [ 5363.375193] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c19d847d000.ost_server_uuid in FULL state after 0 sec [ 5363.613031] LustreError: 157468:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.52@tcp) returned error from blocking AST (req@0000000015f1c781 x1872939184303360 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b4ae65a9/0xe1cdd427c2aa23b4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.52@tcp remote: 0x6b78f1c86fbaa9d expref: 5 pid: 157466 timeout: 5463 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5363.644951] LustreError: lustre-MDT0000: A client on nid 192.168.204.52@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5363.665851] LustreError: 157456:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.52@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b4ae65a9/0xe1cdd427c2aa23b4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.52@tcp remote: 0x6b78f1c86fbaa9d expref: 6 pid: 157466 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5369.551575] Lustre: server umount lustre-OST0000 complete [ 5373.412230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5382.237776] Lustre: server umount lustre-MDT0000 complete [ 5386.235841] LustreError: 157453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175079 with bad export cookie 16270894295994212089 [ 5386.241600] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5386.258467] LustreError: 157453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5396.663168] Lustre: server umount lustre-MDT0001 complete [ 5411.018692] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 5414.674960] Key type lgssc unregistered [ 5415.055762] LNet: 163580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5415.067674] LNetError: 163580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5416.101813] LNet: Removed LNI 192.168.204.152@tcp [ 5417.320441] Key type .llcrypt unregistered [ 5417.322925] Key type ._llcrypt unregistered [ 5441.119976] Key type ._llcrypt registered [ 5441.123919] Key type .llcrypt registered [ 5441.239609] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5442.561555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5442.735119] alg: No test for adler32 (adler32-zlib) [ 5443.891841] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5444.146761] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 5445.823213] Key type lgssc registered [ 5447.219886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5454.461511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5454.480078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.886209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.132119] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5466.080954] LustreError: 164995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.096918] LustreError: 164995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5469.145460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5469.402322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5474.260631] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5481.620651] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5487.880628] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5495.558171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5495.878074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.080120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5503.706585] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5510.540442] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5519.243533] Lustre: Failing over lustre-MDT0000 [ 5519.494864] Lustre: server umount lustre-MDT0000 complete [ 5519.846311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.853238] LustreError: 164990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5519.862951] Lustre: Skipped 1 previous similar message [ 5519.923911] LustreError: 164990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5524.961712] LustreError: 164992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5524.973520] LustreError: 164992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5530.084257] LustreError: 164995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5530.113345] LustreError: 164995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5531.828359] LustreError: 164977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.52@tcp arrived at 1786175224 with bad export cookie 4172135426943422465 [ 5535.202394] LustreError: 164996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.229591] LustreError: 164996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5539.991760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.269568] Lustre: 164308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786175217/real 1786175217] req@0000000073502357 x1872940380483200/t0(0) o400->MGC192.168.204.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786175233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5540.329270] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.357315] LustreError: 164305:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d4c1623c x1872940380487168/t0(0) o250->MGC192.168.204.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5540.835376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.940288] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5545.963381] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5545.999213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5547.686339] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5550.290036] Lustre: lustre-MDT0000: Denying connection for new client 748144ed-3413-4931-ac55-f86b193d935e (at 192.168.204.52@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 5561.314298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.317954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.348789] Lustre: Skipped 2 previous similar messages [ 5564.915050] Lustre: server umount lustre-OST0000 complete [ 5568.133923] LustreError: 168725:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5568.144487] Lustre: 168074:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5568.159234] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5568.183709] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5568.198814] LustreError: 168074:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aba8bf41 x1872940380502400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5568.211780] LustreError: 168074:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5568.223883] LustreError: 168074:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5568.230732] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 5568.462975] Lustre: server umount lustre-MDT0000 complete [ 5571.557099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.581660] LustreError: 166741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.614276] LustreError: 166741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5572.217611] LustreError: 164977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175265 with bad export cookie 4172135426943422570 [ 5572.229078] LustreError: 164977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5572.233719] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.238277] LustreError: 164305:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5574.554308] Lustre: server umount lustre-MDT0001 complete [ 5585.340204] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 5588.295415] Key type lgssc unregistered [ 5588.640382] LNet: 169481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5588.662957] LNetError: 169481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5588.685735] LNet: Removed LNI 192.168.204.152@tcp [ 5590.073234] Key type .llcrypt unregistered [ 5590.075571] Key type ._llcrypt unregistered [ 5614.532720] Key type ._llcrypt registered [ 5614.534885] Key type .llcrypt registered [ 5614.650975] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5615.603928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.736915] alg: No test for adler32 (adler32-zlib) [ 5616.788332] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5617.015501] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 5618.703184] Key type lgssc registered [ 5620.125707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.072172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5629.096382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.462744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.123626] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5640.686187] LustreError: 170910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.721044] LustreError: 170910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5644.365802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.727707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5649.808445] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5658.405426] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5664.070886] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5672.893932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5673.435644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.509130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5679.581843] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5687.106517] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5694.951780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5694.958689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.972271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5694.989140] Lustre: Skipped 1 previous similar message [ 5698.842791] Lustre: server umount lustre-OST0000 complete [ 5705.185632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.193394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5705.209204] Lustre: Skipped 1 previous similar message [ 5705.232245] Lustre: Skipped 2 previous similar messages [ 5710.305823] LustreError: 170910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5710.316348] LustreError: 170910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5711.379538] Lustre: server umount lustre-MDT0000 complete [ 5715.079485] LustreError: 170890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175408 with bad export cookie 12268084194677003692 [ 5715.085184] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.092026] LustreError: 170890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5715.360834] Lustre: server umount lustre-MDT0001 complete [ 5727.180874] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 5730.540490] Key type lgssc unregistered [ 5730.898730] LNet: 174448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5730.905415] LNetError: 174448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5730.926531] LNet: Removed LNI 192.168.204.152@tcp [ 5732.284410] Key type .llcrypt unregistered [ 5732.286423] Key type ._llcrypt unregistered [ 5766.557189] Key type ._llcrypt registered [ 5766.561379] Key type .llcrypt registered [ 5766.829625] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5768.419631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5768.436797] alg: No test for adler32 (adler32-zlib) [ 5769.600083] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5769.965889] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 5771.695209] Key type lgssc registered [ 5773.202414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5781.509852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5781.525953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5782.887222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5788.334955] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5792.231656] LustreError: 176153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.248467] LustreError: 176153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5797.345275] LustreError: 176154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.651393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.948327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5802.775528] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5810.325213] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5816.740255] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5824.332250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5824.655830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.783858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5831.231214] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5840.019386] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5852.644334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5852.657875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.668575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5852.681429] Lustre: Skipped 1 previous similar message [ 5852.697406] Lustre: Skipped 1 previous similar message [ 5857.184456] Lustre: server umount lustre-OST0000 complete [ 5862.885015] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.891802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5862.904578] Lustre: Skipped 1 previous similar message [ 5868.002440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5868.011811] Lustre: Skipped 2 previous similar messages [ 5872.098595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.106458] Lustre: Skipped 1 previous similar message [ 5875.167208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5875.320136] Lustre: server umount lustre-MDT0000 complete [ 5877.218790] LustreError: 176412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.246763] LustreError: 176412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5879.684461] LustreError: 176135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175572 with bad export cookie 7980879882236552860 [ 5879.700211] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5880.052901] Lustre: server umount lustre-MDT0001 complete [ 5893.307068] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 5896.638533] Key type lgssc unregistered [ 5897.019622] LNet: 179693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5897.038227] LNetError: 179693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5897.063745] LNet: Removed LNI 192.168.204.152@tcp [ 5898.361928] Key type .llcrypt unregistered [ 5898.367844] Key type ._llcrypt unregistered [ 5926.714312] Key type ._llcrypt registered [ 5926.718331] Key type .llcrypt registered [ 5926.886515] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 5928.891528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5929.182935] alg: No test for adler32 (adler32-zlib) [ 5930.581305] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 5930.909530] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 5932.593124] Key type lgssc registered [ 5934.237790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5945.778727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5945.796029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5947.192714] Lustre: *** cfs_fail_loc=135, val=0*** [ 5947.262031] LustreError: 181221:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5947.270176] LustreError: 181221:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.152@tcp: cfg command failed: rc = -2 [ 5947.279333] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5947.279333] [ 5947.288656] LustreError: MGC192.168.204.152@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5947.302470] LustreError: 181182:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5947.319186] LustreError: 181182:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5947.329988] LustreError: 181182:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5947.336911] LustreError: 181182:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5947.395555] Lustre: server umount lustre-MDT0000 complete [ 5947.397917] LustreError: 181182:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5954.901611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.265347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5960.093440] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5965.809768] LustreError: 181594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.844197] LustreError: 181594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5968.967039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.444165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.046841] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 5982.105419] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5988.061301] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5990.883424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.888373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5990.900215] Lustre: Skipped 1 previous similar message [ 5990.910761] Lustre: Skipped 1 previous similar message [ 5995.044069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5995.055747] Lustre: Skipped 1 previous similar message [ 5996.202750] Lustre: server umount lustre-MDT0000 complete [ 6000.101716] LustreError: 181594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.115525] LustreError: 181594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6000.334781] LustreError: 181575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175693 with bad export cookie 5619184393825791213 [ 6000.343960] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6000.358346] LustreError: 181575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6000.722768] Lustre: server umount lustre-MDT0001 complete [ 6014.530776] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6018.095930] Key type lgssc unregistered [ 6018.528462] LNet: 183916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.543496] LNetError: 183916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6018.585263] LNet: Removed LNI 192.168.204.152@tcp [ 6019.802486] Key type .llcrypt unregistered [ 6019.819197] Key type ._llcrypt unregistered [ 6043.023417] Key type ._llcrypt registered [ 6043.026528] Key type .llcrypt registered [ 6043.171750] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6043.493175] Key type .llcrypt unregistered [ 6043.495630] Key type ._llcrypt unregistered [ 6064.394309] Key type ._llcrypt registered [ 6064.397823] Key type .llcrypt registered [ 6064.518213] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6066.110623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6066.167053] alg: No test for adler32 (adler32-zlib) [ 6067.752171] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6067.979993] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6069.663186] Key type lgssc registered [ 6070.952802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6081.447662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6081.465692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.991417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6102.111345] LustreError: 186051:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.152@tcp: failed processing log, type 4: rc = -110 [ 6128.800227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6136.360550] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6143.826542] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6151.010399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.568761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.680757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6156.312499] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6156.773393] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6156.805120] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6161.901454] LustreError: 186078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.012993] LustreError: 186077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6164.870954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.266066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6169.743875] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6178.492915] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6183.509537] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6186.591443] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6197.731421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6197.741704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.763886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.319978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.322885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6201.334441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.851157] Lustre: server umount lustre-OST0000 complete [ 6211.555951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.567334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6211.573664] Lustre: Skipped 1 previous similar message [ 6211.581545] Lustre: Skipped 1 previous similar message [ 6216.676874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.683380] Lustre: Skipped 2 previous similar messages [ 6221.797596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6221.806458] Lustre: Skipped 1 previous similar message [ 6223.327132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6223.500795] Lustre: server umount lustre-MDT0000 complete [ 6226.913857] LustreError: 187094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6226.921510] LustreError: 187094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6227.563874] LustreError: 186059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175920 with bad export cookie 15815960197009693397 [ 6227.574973] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.587717] LustreError: 186059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6227.843622] Lustre: server umount lustre-MDT0001 complete [ 6240.379081] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6243.991329] Key type lgssc unregistered [ 6244.293838] LNet: 190018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6244.301241] LNetError: 190018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6244.321575] LNet: Removed LNI 192.168.204.152@tcp [ 6245.379275] Key type .llcrypt unregistered [ 6245.386222] Key type ._llcrypt unregistered [ 6272.890478] Key type ._llcrypt registered [ 6272.895451] Key type .llcrypt registered [ 6273.084156] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6274.423296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6274.450470] alg: No test for adler32 (adler32-zlib) [ 6275.479696] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6275.709581] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6277.447164] Key type lgssc registered [ 6278.584515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.868346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6286.884257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.279907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.402693] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6298.592548] LustreError: 191434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.640380] LustreError: 191434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6302.187440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.489056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6307.910658] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6316.462369] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6322.591696] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6329.613734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6329.873636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.006192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6335.761713] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6343.399265] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6349.120290] Lustre: Failing over lustre-MDT0000 [ 6349.350562] Lustre: server umount lustre-MDT0000 complete [ 6351.437450] LustreError: 191429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.353416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.381030] Lustre: Skipped 2 previous similar messages [ 6356.493178] LustreError: 193514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.509316] LustreError: 193514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6361.608088] LustreError: 193514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.627807] LustreError: 193514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6366.732230] LustreError: 191431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.744680] LustreError: 191431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6367.232048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.322189] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6367.546596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.637316] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6367.671686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6372.554811] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6372.865166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6372.921655] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6372.991454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6377.154386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6387.952813] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6397.273919] Lustre: server umount lustre-OST0000 complete [ 6403.555542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6403.556804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.562347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.578610] Lustre: Skipped 1 previous similar message [ 6408.679706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.683143] Lustre: Skipped 2 previous similar messages [ 6411.848843] Lustre: server umount lustre-MDT0000 complete [ 6413.799222] LustreError: 193514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.822449] LustreError: 193514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6414.816181] Lustre: 190744:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786176091/real 1786176091] req@0000000029d1d3f5 x1872941251875968/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786176107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6414.847693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.379854] LustreError: 195028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176109 with bad export cookie 1659885223315802440 [ 6416.382092] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6416.402383] LustreError: 195028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6416.694584] Lustre: server umount lustre-MDT0001 complete [ 6429.829200] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6432.564966] Key type lgssc unregistered [ 6432.841974] LNet: 196433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6432.848862] LNetError: 196433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6432.866854] LNet: Removed LNI 192.168.204.152@tcp [ 6433.688580] Key type .llcrypt unregistered [ 6433.691366] Key type ._llcrypt unregistered [ 6460.090760] Key type ._llcrypt registered [ 6460.093428] Key type .llcrypt registered [ 6460.210372] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6461.889927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6462.273433] alg: No test for adler32 (adler32-zlib) [ 6463.340868] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6463.594979] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6465.359198] Key type lgssc registered [ 6466.684894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6475.188599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6475.220590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.872814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.705990] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6487.013179] LustreError: 197849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.045205] LustreError: 197849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6490.155501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.505767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6496.025447] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6503.920976] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6509.608964] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6516.128301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6516.583980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6521.728187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6523.010153] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6530.830253] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6533.345212] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6539.736840] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6556.492550] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6561.760460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6561.765344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6561.793155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6561.829799] Lustre: Skipped 1 previous similar message [ 6566.884660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6566.891653] Lustre: Skipped 1 previous similar message [ 6567.558645] Lustre: server umount lustre-OST0000 complete [ 6572.000330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.019568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6572.029042] Lustre: Skipped 2 previous similar messages [ 6577.122345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6577.127971] Lustre: Skipped 1 previous similar message [ 6577.540670] Lustre: server umount lustre-MDT0000 complete [ 6581.231111] LustreError: 197828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176274 with bad export cookie 5680445719023325017 [ 6581.244777] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6581.567104] Lustre: server umount lustre-MDT0001 complete [ 6593.625695] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6596.409494] Key type lgssc unregistered [ 6596.696844] LNet: 201717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6596.719905] LNetError: 201717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6596.738155] LNet: Removed LNI 192.168.204.152@tcp [ 6597.585411] Key type .llcrypt unregistered [ 6597.589743] Key type ._llcrypt unregistered [ 6616.606405] Key type ._llcrypt registered [ 6616.611772] Key type .llcrypt registered [ 6616.752660] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6618.274017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6618.308981] alg: No test for adler32 (adler32-zlib) [ 6619.338519] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6619.556794] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6621.215173] Key type lgssc registered [ 6622.243463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6629.003773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6629.024298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.405849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.297395] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6640.617683] LustreError: 202985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.646294] LustreError: 202985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6641.901094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.199260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6647.041699] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6654.142704] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6659.776331] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6667.056871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.514046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.334902] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6675.716149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6681.973191] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6686.827897] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6698.207832] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6707.631755] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6713.370514] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6725.089784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.095939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.114749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.137810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6727.141214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.150899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.258896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6732.269482] Lustre: Skipped 2 previous similar messages [ 6737.383379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6737.394815] Lustre: Skipped 1 previous similar message [ 6738.399358] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6738.607853] Lustre: server umount lustre-OST0000 complete [ 6744.545573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6744.556358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6744.584524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6747.622753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.737330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6752.740338] Lustre: Skipped 3 previous similar messages [ 6757.343319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6757.579233] Lustre: server umount lustre-MDT0000 complete [ 6757.857027] LustreError: 202985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.896753] LustreError: 202985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6760.879199] LustreError: 206062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176453 with bad export cookie 4396231237457397308 [ 6760.879612] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.892052] LustreError: 206062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6761.277287] Lustre: server umount lustre-MDT0001 complete [ 6772.349979] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6775.971721] Key type lgssc unregistered [ 6776.216897] LNet: 207961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6776.232433] LNetError: 207961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6776.251595] LNet: Removed LNI 192.168.204.152@tcp [ 6777.104735] Key type .llcrypt unregistered [ 6777.108817] Key type ._llcrypt unregistered [ 6802.219467] Key type ._llcrypt registered [ 6802.221744] Key type .llcrypt registered [ 6802.316779] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6803.315917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6803.524355] alg: No test for adler32 (adler32-zlib) [ 6804.599346] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6804.829525] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6806.512166] Key type lgssc registered [ 6807.755497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6815.908736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6815.935273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.435932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.285948] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6828.008012] LustreError: 209377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.034988] LustreError: 209377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6830.375300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.693028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6834.959396] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6841.771352] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6847.524570] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6854.582911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.899250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.984207] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6863.095645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6871.518120] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6888.943312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.943581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6888.970958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6888.993714] Lustre: Skipped 1 previous similar message [ 6894.101760] Lustre: server umount lustre-OST0000 complete [ 6899.170439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.176900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6899.199406] Lustre: Skipped 2 previous similar messages [ 6899.213337] Lustre: Skipped 1 previous similar message [ 6904.294323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6904.299265] Lustre: Skipped 1 previous similar message [ 6905.145534] Lustre: server umount lustre-MDT0000 complete [ 6908.401170] LustreError: 209376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.437200] LustreError: 209376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6909.225668] LustreError: 209357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176602 with bad export cookie 5751622721241882220 [ 6909.238465] LustreError: 209357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6909.238992] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6909.515579] Lustre: server umount lustre-MDT0001 complete [ 6920.500207] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 6923.975240] Key type lgssc unregistered [ 6924.416370] LNet: 213063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6924.427645] LNetError: 213063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6924.440227] LNet: Removed LNI 192.168.204.152@tcp [ 6925.317353] Key type .llcrypt unregistered [ 6925.321771] Key type ._llcrypt unregistered [ 6947.552655] Key type ._llcrypt registered [ 6947.555284] Key type .llcrypt registered [ 6947.650526] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 6949.063639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6949.223359] alg: No test for adler32 (adler32-zlib) [ 6950.531916] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 6950.802549] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 6952.511230] Key type lgssc registered [ 6954.220194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6962.193599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6962.239567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.668695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6969.153344] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6973.928644] LustreError: 214333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.955557] LustreError: 214333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6979.047791] LustreError: 214334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.292881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.579148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6984.659054] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 6993.486564] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6999.055281] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7005.474732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.049338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.078794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7012.626316] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7020.750462] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7039.970126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7039.973361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.981042] LustreError: Skipped 1 previous similar message [ 7040.004622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.405521] Lustre: server umount lustre-OST0000 complete [ 7045.089876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7045.092582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7045.098842] Lustre: Skipped 2 previous similar messages [ 7045.113719] Lustre: Skipped 2 previous similar messages [ 7050.210640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.222703] Lustre: Skipped 1 previous similar message [ 7051.218501] Lustre: server umount lustre-MDT0000 complete [ 7054.641148] LustreError: 214314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176747 with bad export cookie 15837974889487638592 [ 7054.651160] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7054.658799] LustreError: 214314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7055.035206] Lustre: server umount lustre-MDT0001 complete [ 7067.256603] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7070.828226] Key type lgssc unregistered [ 7071.094585] LNet: 218020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7071.102640] LNetError: 218020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7071.116654] LNet: Removed LNI 192.168.204.152@tcp [ 7071.943577] Key type .llcrypt unregistered [ 7071.945757] Key type ._llcrypt unregistered [ 7094.424899] Key type ._llcrypt registered [ 7094.427306] Key type .llcrypt registered [ 7094.532226] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7095.298541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7095.424576] alg: No test for adler32 (adler32-zlib) [ 7096.490139] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7096.714774] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7098.359239] Key type lgssc registered [ 7099.868327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7108.909937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7108.935548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7110.492385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7114.886443] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7120.869440] LustreError: 219288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7120.907728] LustreError: 219288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7125.269342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7125.718913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7131.965966] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7140.077410] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7146.730700] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7153.804803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.140282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7155.195316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7160.910986] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7169.194349] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7179.973424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7180.056766] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7180.313580] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7187.684421] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7196.238194] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7197.956620] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7210.069617] Lustre: Permanently deactivating lustre-OST0001 [ 7210.107316] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7215.704854] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7224.375623] Lustre: Permanently reactivating lustre-OST0001 [ 7224.393337] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7224.399255] Lustre: Skipped 2 previous similar messages [ 7234.136938] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.153398] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7234.161898] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7234.187992] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7234.314890] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7236.863548] Lustre: server umount lustre-OST0001 complete [ 7237.102255] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.153577] Lustre: Skipped 2 previous similar messages [ 7237.158043] LustreError: 221783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.248831] LustreError: 221783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7242.210844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.218405] LustreError: 221783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7242.218763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7242.220249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7242.230820] Lustre: Skipped 1 previous similar message [ 7242.255359] LustreError: 221783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7247.331916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.337562] Lustre: Skipped 2 previous similar messages [ 7247.339974] LustreError: 221113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.384360] LustreError: 221113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7247.810494] Lustre: server umount lustre-OST0000 complete [ 7252.452376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.458745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7252.471628] Lustre: Skipped 1 previous similar message [ 7252.482041] Lustre: Skipped 1 previous similar message [ 7257.569604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7258.391635] Lustre: server umount lustre-MDT0000 complete [ 7262.692991] LustreError: 220017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7262.718221] LustreError: 220017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7262.824653] LustreError: 219268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176955 with bad export cookie 5807886229941400090 [ 7262.833639] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7262.840261] LustreError: 219268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7263.248344] Lustre: server umount lustre-MDT0001 complete [ 7275.134286] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7278.749722] Key type lgssc unregistered [ 7279.122273] LNet: 224342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7279.141328] LNetError: 224342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7279.174613] LNet: Removed LNI 192.168.204.152@tcp [ 7280.678628] Key type .llcrypt unregistered [ 7280.680713] Key type ._llcrypt unregistered [ 7325.310583] Key type ._llcrypt registered [ 7325.312250] Key type .llcrypt registered [ 7325.384187] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7327.127804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7327.500412] alg: No test for adler32 (adler32-zlib) [ 7328.572644] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7328.828070] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7330.487283] Key type lgssc registered [ 7332.309571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7341.610405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7341.639950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.969909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7342.999267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7343.214550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7348.135397] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7357.280504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.352574] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7357.373437] Lustre: 226549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7357.610232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7361.865867] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7369.810922] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7375.998315] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7382.991859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7383.155816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7383.440812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.667988] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7391.538554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7398.826408] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.488488] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7408.156391] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7417.553458] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7426.837715] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7439.008846] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7439.014429] Lustre: Skipped 1 previous similar message [ 7455.447106] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7455.451895] Lustre: Skipped 1 previous similar message [ 7495.576817] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7495.581813] Lustre: Skipped 4 previous similar messages [ 7563.749976] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7563.758970] Lustre: Skipped 7 previous similar messages [ 7586.277893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7586.279231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7586.290061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.308293] Lustre: Skipped 1 previous similar message [ 7586.319916] LustreError: Skipped 1 previous similar message [ 7588.339891] Lustre: server umount lustre-OST0000 complete [ 7593.442762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7593.451800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.470304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.475522] Lustre: Skipped 1 previous similar message [ 7596.517460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7596.519493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7599.653853] Lustre: server umount lustre-MDT0000 complete [ 7604.250860] LustreError: 227292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177297 with bad export cookie 5944661673727657456 [ 7604.251510] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7604.266686] LustreError: 227292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7604.615889] Lustre: server umount lustre-MDT0001 complete [ 7617.554984] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7620.529478] Key type lgssc unregistered [ 7620.946915] LNet: 231357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7620.953997] LNetError: 231357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7620.969879] LNet: Removed LNI 192.168.204.152@tcp [ 7622.392501] Key type .llcrypt unregistered [ 7622.397828] Key type ._llcrypt unregistered [ 7647.856828] Key type ._llcrypt registered [ 7647.860049] Key type .llcrypt registered [ 7648.040559] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7649.313136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7649.373721] alg: No test for adler32 (adler32-zlib) [ 7650.342735] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7650.531325] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7652.183815] Key type lgssc registered [ 7652.906683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7660.613996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7660.628827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.913939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7667.624666] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7671.270247] LustreError: 232752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.278717] LustreError: 232752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7676.387548] LustreError: 232751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7680.926857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.465252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7685.385587] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7699.707923] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7709.959560] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7720.293366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.990985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7729.241788] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_default_debug -1 all [ 7731.247959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7739.851100] Lustre: DEBUG MARKER: oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7760.300990] Lustre: server umount lustre-OST0000 complete [ 7767.012854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.017622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.028176] Lustre: Skipped 1 previous similar message [ 7767.036992] Lustre: Skipped 1 previous similar message [ 7772.134194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.149669] Lustre: Skipped 1 previous similar message [ 7774.214800] Lustre: server umount lustre-MDT0000 complete [ 7777.120429] Lustre: 232063:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786177454/real 1786177454] req@00000000a6389ed7 x1872942693657216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786177470 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7777.154350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.168308] LustreError: 232751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.188579] LustreError: 232751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7777.918625] LustreError: 235335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177470 with bad export cookie 6056630318825131600 [ 7777.922163] LustreError: MGC192.168.204.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7777.930523] LustreError: 235335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7778.287245] Lustre: server umount lustre-MDT0001 complete [ 7790.852557] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 7793.747901] Key type lgssc unregistered [ 7794.003218] LNet: 236453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7794.011577] LNetError: 236453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7794.028829] LNet: Removed LNI 192.168.204.152@tcp [ 7795.078632] Key type .llcrypt unregistered [ 7795.084349] Key type ._llcrypt unregistered [ 7822.902335] Key type ._llcrypt registered [ 7822.905520] Key type .llcrypt registered [ 7823.020178] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 7824.331388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7824.344119] alg: No test for adler32 (adler32-zlib) [ 7825.620485] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 7825.904595] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 7827.767369] Key type lgssc registered [ 7829.222064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8099.808930] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8103.143321] Key type lgssc unregistered [ 8103.402482] LNet: 238490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8103.412003] LNetError: 238490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8103.425819] LNet: Removed LNI 192.168.204.152@tcp [ 8104.599693] Key type .llcrypt unregistered [ 8104.607098] Key type ._llcrypt unregistered [ 8131.555762] Key type ._llcrypt registered [ 8131.563761] Key type .llcrypt registered [ 8131.700466] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing set_hostid [ 8148.664134] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing load_modules_local [ 8149.881988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8149.912651] alg: No test for adler32 (adler32-zlib) [ 8150.918958] Lustre: Lustre: Build Version: 2.17.56_50_gc905d6b [ 8151.149325] LNet: Added LNI 192.168.204.152@tcp [8/256/0/180] [ 8152.823244] Key type lgssc registered [ 8153.968670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8180.413916] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7849 sec ========== 04:31:12 (1786177872) [ 8182.006656] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:31:14 (1786177874) === [ 8185.020927] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:31:17 (1786177877) === [ 8223.494179] Lustre: DEBUG MARKER: oleg452-server.virtnet: executing unload_modules_local [ 8226.329981] Key type lgssc unregistered [ 8226.703270] LNet: 243728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8226.710802] LNetError: 243728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8226.734680] LNet: Removed LNI 192.168.204.152@tcp [ 8227.658773] Key type .llcrypt unregistered [ 8227.671847] Key type ._llcrypt unregistered