[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516957372 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002250] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008127] LSM: Security Framework initializing [ 0.009049] Yama: becoming mindful. [ 0.010033] SELinux: Initializing. [ 0.011054] *** VALIDATE selinux *** [ 0.018482] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022663] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024027] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026095] *** VALIDATE tmpfs *** [ 0.027254] *** VALIDATE proc *** [ 0.028267] *** VALIDATE cgroup *** [ 0.029010] *** VALIDATE cgroup2 *** [ 0.030287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033034] Spectre V2 : User space: Vulnerable [ 0.034011] Speculative Store Bypass: Vulnerable [ 0.037414] debug: unmapping init [mem 0xffffffff95c59000-0xffffffff95c60fff] [ 0.040179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043009] ... bit width: 48 [ 0.043996] ... generic registers: 4 [ 0.044018] ... value mask: 0000ffffffffffff [ 0.045013] ... max period: 00007fffffffffff [ 0.046013] ... fixed-purpose events: 3 [ 0.047010] ... event mask: 000000070000000f [ 0.048335] rcu: Hierarchical SRCU implementation. [ 0.050613] smp: Bringing up secondary CPUs ... [ 0.051514] x86: Booting SMP configuration: [ 0.052021] .... node #0, CPUs: #1 #2 #3 [ 0.057557] smp: Brought up 1 node, 4 CPUs [ 0.059017] smpboot: Max logical packages: 1 [ 0.060018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.116802] node 0 deferred pages initialised in 53ms [ 0.120084] devtmpfs: initialized [ 0.121207] x86/mm: Memory block size: 128MB [ 0.123486] gcov: version magic: 0x41383552 [ 0.125358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.130265] pinctrl core: initialized pinctrl subsystem [ 0.131120] [ 0.131393] ************************************************************* [ 0.133013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135007] ** ** [ 0.136007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137010] ** ** [ 0.139007] ** This means that this kernel is built to expose internal ** [ 0.140011] ** IOMMU data structures, which may compromise security on ** [ 0.143010] ** your system. ** [ 0.145011] ** ** [ 0.148014] ** If you see this message and you are not debugging the ** [ 0.151013] ** kernel, report this immediately to your vendor! ** [ 0.153010] ** ** [ 0.156019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158013] ************************************************************* [ 0.161653] NET: Registered protocol family 16 [ 0.164883] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180011] cpuidle: using governor menu [ 0.182156] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185421] PCI: Using configuration type 1 for base access [ 0.187121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204024] cryptd: max_cpu_qlen set to 1000 [ 0.206253] ACPI: Added _OSI(Module Device) [ 0.207016] ACPI: Added _OSI(Processor Device) [ 0.207933] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214567] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222015] ACPI: Interpreter enabled [ 0.225088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228101] ACPI: Using IOAPIC for interrupt routing [ 0.230131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.233758] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241328] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252133] acpiphp: Slot [2] registered [ 0.252991] acpiphp: Slot [5] registered [ 0.254059] acpiphp: Slot [6] registered [ 0.254829] acpiphp: Slot [7] registered [ 0.256214] acpiphp: Slot [8] registered [ 0.257134] acpiphp: Slot [9] registered [ 0.259092] acpiphp: Slot [10] registered [ 0.261124] acpiphp: Slot [3] registered [ 0.262089] acpiphp: Slot [4] registered [ 0.263062] acpiphp: Slot [11] registered [ 0.264105] acpiphp: Slot [12] registered [ 0.266113] acpiphp: Slot [13] registered [ 0.266995] acpiphp: Slot [14] registered [ 0.267057] acpiphp: Slot [15] registered [ 0.268082] acpiphp: Slot [16] registered [ 0.269077] acpiphp: Slot [17] registered [ 0.271070] acpiphp: Slot [18] registered [ 0.272000] acpiphp: Slot [19] registered [ 0.272000] acpiphp: Slot [20] registered [ 0.273079] acpiphp: Slot [21] registered [ 0.274058] acpiphp: Slot [22] registered [ 0.275073] acpiphp: Slot [23] registered [ 0.276084] acpiphp: Slot [24] registered [ 0.278082] acpiphp: Slot [25] registered [ 0.279120] acpiphp: Slot [26] registered [ 0.281125] acpiphp: Slot [27] registered [ 0.283124] acpiphp: Slot [28] registered [ 0.284125] acpiphp: Slot [29] registered [ 0.286076] acpiphp: Slot [30] registered [ 0.287093] acpiphp: Slot [31] registered [ 0.288059] PCI host bridge to bus 0000:00 [ 0.289019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.290022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.296017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305058] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.308489] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321549] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.324027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.328017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.331508] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.333499] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.335036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338847] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.355020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.367264] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.376021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.383020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.398018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.410728] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.416019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.423021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.437022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.446276] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.463017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.478021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.489058] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.494017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.500019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.513017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.524015] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.528013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.532000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.543024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.552448] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.559021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.565016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.582020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.593915] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.596354] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.598382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.601417] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.603280] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.608113] iommu: Default domain type: Passthrough [ 0.609412] SCSI subsystem initialized [ 0.610152] ACPI: bus type USB registered [ 0.611135] usbcore: registered new interface driver usbfs [ 0.613112] usbcore: registered new interface driver hub [ 0.615125] usbcore: registered new device driver usb [ 0.616094] pps_core: LinuxPPS API ver. 1 registered [ 0.617007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.619075] PTP clock support registered [ 0.622084] EDAC MC: Ver: 3.0.0 [ 0.623598] PCI: Using ACPI for IRQ routing [ 0.624680] NetLabel: Initializing [ 0.625013] NetLabel: domain hash size = 128 [ 0.626014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.627089] NetLabel: unlabeled traffic allowed by default [ 0.629107] vgaarb: loaded [ 0.630250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.632016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.638000] clocksource: Switched to clocksource kvm-clock [ 0.731475] VFS: Disk quotas dquot_6.6.0 [ 0.733364] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.735387] *** VALIDATE ramfs *** [ 0.736351] *** VALIDATE hugetlbfs *** [ 0.737951] pnp: PnP ACPI init [ 0.739964] pnp: PnP ACPI: found 6 devices [ 0.752645] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.754582] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.756127] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.758097] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.760555] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.763308] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.765993] NET: Registered protocol family 2 [ 0.768507] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.773224] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.776549] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.781827] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.785328] TCP: Hash tables configured (established 65536 bind 65536) [ 0.788233] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.791253] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.793965] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.796717] NET: Registered protocol family 1 [ 0.798966] RPC: Registered named UNIX socket transport module. [ 0.801072] RPC: Registered udp transport module. [ 0.804399] RPC: Registered tcp transport module. [ 0.805781] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.807508] NET: Registered protocol family 44 [ 0.809097] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.812484] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.814542] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.817375] PCI: CLS 0 bytes, default 64 [ 0.819466] Unpacking initramfs... [ 2.187129] debug: unmapping init [mem 0xffff934ebcc54000-0xffff934ebffbffff] [ 2.191463] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.193659] software IO TLB: mapped [mem 0x00000000a0a00000-0x00000000a4a00000] (64MB) [ 2.196369] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.644705] Initialise system trusted keyrings [ 2.645819] Key type blacklist registered [ 2.646967] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.654210] zbud: loaded [ 2.657379] *** VALIDATE nfs *** [ 2.658509] *** VALIDATE nfs4 *** [ 2.659498] pstore: using deflate compression [ 2.661966] Platform Keyring initialized [ 2.746829] NET: Registered protocol family 38 [ 2.748277] Key type asymmetric registered [ 2.749285] Asymmetric key parser 'x509' registered [ 2.750667] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.752603] io scheduler mq-deadline registered [ 2.753693] io scheduler kyber registered [ 2.754830] io scheduler bfq registered [ 2.756055] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.758124] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.759978] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.761831] ACPI: Power Button [PWRF] [ 2.765988] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.771465] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.779518] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.784437] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.794029] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.823238] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.852035] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.857184] Non-volatile memory driver v1.3 [ 2.858930] Linux agpgart interface v0.103 [ 2.890959] virtio_blk virtio1: [vda] 145944 512-byte logical blocks (74.7 MB/71.3 MiB) [ 2.893200] vda: detected capacity change from 0 to 74723328 [ 2.905702] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.907858] vdb: detected capacity change from 0 to 1073741824 [ 2.919781] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.922397] vdc: detected capacity change from 0 to 2621440000 [ 2.933372] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.935680] vdd: detected capacity change from 0 to 2621440000 [ 2.946964] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.948851] vde: detected capacity change from 0 to 4294967296 [ 2.959294] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.961179] vdf: detected capacity change from 0 to 4294967296 [ 2.965227] libphy: Fixed MDIO Bus: probed [ 2.976153] usbcore: registered new interface driver usbserial_generic [ 2.979798] usbserial: USB Serial support registered for generic [ 2.983218] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.986183] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.987311] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.989031] mousedev: PS/2 mouse device common for all mice [ 2.991308] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.993330] rtc_cmos 00:05: RTC can wake from S4 [ 2.996403] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.998673] rtc_cmos 00:05: registered as rtc0 [ 3.002106] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.002806] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.004212] intel_pstate: CPU model not supported [ 3.008980] hid: raw HID events driver (C) Jiri Kosina [ 3.010365] usbcore: registered new interface driver usbhid [ 3.011568] usbhid: USB HID core driver [ 3.012656] drop_monitor: Initializing network drop monitor service [ 3.014542] Initializing XFRM netlink socket [ 3.016026] NET: Registered protocol family 10 [ 3.018075] Segment Routing with IPv6 [ 3.019083] NET: Registered protocol family 17 [ 3.020821] mpls_gso: MPLS GSO support [ 3.024926] RAS: Correctable Errors collector initialized. [ 3.027101] AVX version of gcm_enc/dec engaged. [ 3.029217] AES CTR mode by8 optimization enabled [ 3.102109] sched_clock: Marking stable (3102057921, 0)->(3942014157, -839956236) [ 3.105244] registered taskstats version 1 [ 3.106973] Loading compiled-in X.509 certificates [ 3.108721] zswap: loaded using pool lzo/zbud [ 3.129926] Key type big_key registered [ 3.139654] Key type encrypted registered [ 3.140632] ima: No TPM chip found, activating TPM-bypass! [ 3.141752] ima: Allocated hash algorithm: sha1 [ 3.143335] ima: No architecture policies found [ 3.144477] evm: Initialising EVM extended attributes: [ 3.145481] evm: security.selinux [ 3.146176] evm: security.ima [ 3.146706] evm: security.capability [ 3.147433] evm: HMAC attrs: 0x1 [ 3.149050] rtc_cmos 00:05: setting system clock to 2026-08-14 23:51:29 UTC (1786751489) [ 3.152926] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 3.155480] debug: unmapping init [mem 0xffffffff95982000-0xffffffff95c58fff] [ 3.165077] Write protecting the kernel read-only data: 28672k [ 3.168225] debug: unmapping init [mem 0xffffffff94003000-0xffffffff941fffff] [ 3.170913] debug: unmapping init [mem 0xffffffff94914000-0xffffffff949fffff] [ 3.205534] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.213537] systemd[1]: Detected virtualization kvm. [ 3.215716] systemd[1]: Detected architecture x86-64. [ 3.218406] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.243388] systemd[1]: No hostname configured. [ 3.245244] systemd[1]: Set hostname to . [ 3.247636] random: systemd: uninitialized urandom read (16 bytes read) [ 3.250773] systemd[1]: Initializing machine ID from random generator. [ 3.307737] random: ln: uninitialized urandom read (6 bytes read) [ 3.405066] random: systemd: uninitialized urandom read (16 bytes read) [ 3.407499] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.412808] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.418475] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.023340] device-mapper: uevent: version 1.0.3 [ 4.025742] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.689081] virtio_net virtio0 ens2: renamed from eth0 [ 4.739094] random: fast init done [ 4.827248] scsi host0: ata_piix [ 4.843267] scsi host1: ata_piix [ 4.845117] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.847486] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.471756] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.652077] random: crng init done [ 9.653387] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.056887] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.237712] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.450196] SELinux: Disabled at runtime. [ 11.502417] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.507420] systemd[1]: Detected virtualization kvm. [ 11.508535] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.955686] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.958479] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.962666] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.965525] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.968416] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.975339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.978765] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.079062] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.129717] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.231373] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.455530] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.996256] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.079219] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root sup[ 21.441772] Key type dns_resolver registered port (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 22.459630] NFS: Registering the id_resolver key type [ 22.462340] Key type id_resolver registered [ 22.463859] Key type id_legacy registered [ *] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg154-server login: [ 90.572431] libcfs: loading out-of-tree module taints kernel. [ 90.674146] Key type ._llcrypt registered [ 90.686034] Key type .llcrypt registered [ 90.798835] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_hostid [ 112.381503] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 114.186782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.253927] alg: No test for adler32 (adler32-zlib) [ 116.621324] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 118.549266] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 120.719258] Key type lgssc registered [ 123.929979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.954377] hrtimer: interrupt took 2868447 ns [ 144.097109] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 194.010501] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 207.621149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 207.672717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.094753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.163697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.412138] Lustre: lustre-MDT0000: new disk, initializing [ 209.620513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.650550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.302399] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.886760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.074662] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.114796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.119302] Lustre: Skipped 1 previous similar message [ 230.247196] Lustre: lustre-MDT0001: new disk, initializing [ 230.362298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.425801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.435238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 234.262286] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.627607] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 251.947639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.463821] Lustre: lustre-OST0000: new disk, initializing [ 252.469631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 252.483436] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.562230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.112219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 256.117265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.256530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.676930] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.750444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.951530] Lustre: lustre-OST0001: new disk, initializing [ 273.954882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.959789] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 274.033633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 281.559919] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.616947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.623275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.711210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 292.672672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.311787] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 307.746338] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing check_logdir /tmp/testlogs/ [ 312.786294] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing yml_node [ 319.316619] Lustre: DEBUG MARKER: Client: 2.17.57.4 [ 321.961687] Lustre: DEBUG MARKER: MDS: 2.17.57.4 [ 324.073573] Lustre: DEBUG MARKER: OSS: 2.17.57.4 [ 326.302985] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 19:56:50 EDT 2026 [ 346.004183] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 347.811692] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 363.491461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.497976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.504032] Lustre: Skipped 1 previous similar message [ 363.516319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 368.617268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.636248] Lustre: Skipped 6 previous similar messages [ 369.777865] Lustre: server umount lustre-MDT0000 complete [ 373.642723] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751859 with bad export cookie 38039890941856774 [ 373.643662] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.657160] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 373.738174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.744315] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.754315] Lustre: Skipped 3 previous similar messages [ 378.857138] LustreError: 9538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.859117] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.871442] LustreError: 9538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 378.898029] Lustre: Skipped 2 previous similar messages [ 380.106721] Lustre: server umount lustre-MDT0001 complete [ 384.544635] Lustre: server umount lustre-OST0000 complete [ 389.566766] Lustre: server umount lustre-OST0001 complete [ 397.600303] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_hostid [ 409.862548] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 451.389694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.806657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 451.858176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.022097] Lustre: lustre-MDT0000: new disk, initializing [ 452.151312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.190632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 456.497667] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 467.178526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.380079] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 467.482522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 467.491406] Lustre: Skipped 1 previous similar message [ 467.683096] Lustre: lustre-MDT0001: new disk, initializing [ 467.853098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 467.900818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 467.918408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 472.873582] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 483.828599] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.756069] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 497.488484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.729451] Lustre: lustre-OST0000: new disk, initializing [ 497.733345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 497.738376] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 497.816914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.742027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 499.755589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 499.878880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.993653] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 512.061183] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 516.355843] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 516.543727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 520.616461] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 520.795989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 525.294552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.301248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.321171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.340266] Lustre: Skipped 1 previous similar message [ 525.364235] Lustre: Skipped 1 previous similar message [ 529.318806] Lustre: server umount lustre-OST0000 complete [ 534.497149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.512660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.526187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.295387] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 547.715631] Lustre: server umount lustre-MDT0000 complete [ 549.877738] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.911517] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 551.994375] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752038 with bad export cookie 38039890941858307 [ 551.998075] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.003426] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 552.466390] Lustre: server umount lustre-MDT0001 complete [ 555.945096] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:00:41 (1786752041) [ 561.710788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.027382] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.114776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.185190] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 567.265908] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.391678] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.192803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.177843] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 589.672876] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 595.438760] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 602.262746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 607.915271] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 614.726621] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 627.184357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 627.188371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.204282] Lustre: Skipped 1 previous similar message [ 627.220675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.225277] Lustre: Skipped 6 previous similar messages [ 627.914835] Lustre: server umount lustre-OST0000 complete [ 635.363193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.383905] Lustre: Skipped 1 previous similar message [ 646.113283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 646.593234] Lustre: server umount lustre-MDT0000 complete [ 646.625123] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752116/real 1786752116] req@ffff934f3741dc00 x1873544850088192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786752132 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 646.672254] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.403617] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752137 with bad export cookie 38039890941859189 [ 651.411724] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.419386] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 651.799158] Lustre: server umount lustre-MDT0001 complete [ 666.938584] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 670.044860] Key type lgssc unregistered [ 670.582661] LNet: 22869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.587073] LNetError: 22869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.653374] LNet: Removed LNI 192.168.201.154@tcp [ 673.321305] Key type .llcrypt unregistered [ 673.326621] Key type ._llcrypt unregistered [ 699.742164] Key type ._llcrypt registered [ 699.745518] Key type .llcrypt registered [ 699.881848] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 701.122327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 701.167522] alg: No test for adler32 (adler32-zlib) [ 702.284152] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 702.546827] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 704.288699] Key type lgssc registered [ 705.436080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.418975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.446697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.014787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.294825] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 725.476902] LustreError: 24283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.501478] LustreError: 24283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 727.363228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.855459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 732.127644] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 738.133405] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 743.352906] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 751.761736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 752.184817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.386890] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 760.376495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 766.059368] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 783.591736] Lustre: server umount lustre-OST0000 complete [ 789.474475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.481574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.500906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.025727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.032139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.134243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.136942] Lustre: Skipped 1 previous similar message [ 797.235001] Lustre: server umount lustre-MDT0000 complete [ 800.225425] Lustre: 23594:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786752270/real 1786752270] req@00000000f6cf5318 x1873545462314496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786752286 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 800.245869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.263092] LustreError: 25012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 800.273494] LustreError: 25012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 800.784583] LustreError: 24264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752287 with bad export cookie 14227793438912938034 [ 800.786906] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.805391] LustreError: 24264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 801.147989] Lustre: server umount lustre-MDT0001 complete [ 812.138364] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 815.688528] Key type lgssc unregistered [ 816.029798] LNet: 28260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.065203] LNetError: 28260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.159376] LNet: Removed LNI 192.168.201.154@tcp [ 818.230691] Key type .llcrypt unregistered [ 818.233350] Key type ._llcrypt unregistered [ 843.769782] Key type ._llcrypt registered [ 843.772256] Key type .llcrypt registered [ 843.871564] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 845.182914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.240249] alg: No test for adler32 (adler32-zlib) [ 846.245691] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 846.426820] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 848.095200] Key type lgssc registered [ 849.630228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 857.015815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 857.038330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.655240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.126609] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 867.819347] LustreError: 29673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.839454] LustreError: 29673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 870.897239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.182749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.780084] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 883.938690] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 891.015469] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 908.231804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 908.644119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.694251] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 925.503207] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 928.128051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 941.539205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 941.542192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.572814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.585822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.586752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.807683] Lustre: server umount lustre-OST0000 complete [ 948.708295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.718312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.722875] Lustre: Skipped 1 previous similar message [ 953.314590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 953.321725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.333128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.879901] Lustre: server umount lustre-MDT0000 complete [ 958.947740] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.973954] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 960.481089] LustreError: 31862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752446 with bad export cookie 8880049560542368792 [ 960.487955] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.497993] LustreError: 31862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 960.939686] Lustre: server umount lustre-MDT0001 complete [ 972.513066] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 976.214773] Key type lgssc unregistered [ 976.467621] LNet: 33579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.472816] LNetError: 33579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.511509] LNet: Removed LNI 192.168.201.154@tcp [ 978.380844] Key type .llcrypt unregistered [ 978.383546] Key type ._llcrypt unregistered [ 1005.294813] Key type ._llcrypt registered [ 1005.296279] Key type .llcrypt registered [ 1005.467404] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 1006.993294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.111443] alg: No test for adler32 (adler32-zlib) [ 1008.192852] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1008.465071] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 1010.143174] Key type lgssc registered [ 1011.658379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.850326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.866916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.504095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.183836] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1032.677459] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.727292] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1037.796821] LustreError: 34996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.494833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.973525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.811294] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1053.561337] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.000910] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1068.953411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1069.209812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.733674] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1083.383588] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1092.807059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1103.328131] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.330912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.344029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.344777] Lustre: Skipped 1 previous similar message [ 1108.449884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.463215] Lustre: Skipped 2 previous similar messages [ 1109.473653] Lustre: server umount lustre-OST0000 complete [ 1115.617697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1115.634400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.651579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1118.694522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1118.700560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.493763] Lustre: server umount lustre-MDT0000 complete [ 1123.808903] LustreError: 34990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1123.829654] LustreError: 34990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1124.368165] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752610 with bad export cookie 6056377099861016967 [ 1124.389771] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.633327] Lustre: server umount lustre-MDT0001 complete [ 1135.660378] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 1140.251474] Key type lgssc unregistered [ 1140.478306] LNet: 38553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.483468] LNetError: 38553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.494829] LNet: Removed LNI 192.168.201.154@tcp [ 1141.384754] Key type .llcrypt unregistered [ 1141.386328] Key type ._llcrypt unregistered [ 1169.453394] Key type ._llcrypt registered [ 1169.458340] Key type .llcrypt registered [ 1169.586880] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 1171.127320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.148543] alg: No test for adler32 (adler32-zlib) [ 1172.407760] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1172.632241] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 1174.327191] Key type lgssc registered [ 1175.678278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1183.480403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1183.522926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.994642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.682264] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1195.506174] LustreError: 39973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.545688] LustreError: 39973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1197.538500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.038054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1203.156792] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1210.655763] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.457404] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.008883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.347925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.643874] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1230.534941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1237.848925] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1248.232868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.248618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1248.255226] Lustre: Skipped 1 previous similar message [ 1250.189360] Lustre: server umount lustre-OST0000 complete [ 1269.729712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1269.740929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.761091] Lustre: Skipped 1 previous similar message [ 1269.766309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.805554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.805680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.954857] Lustre: server umount lustre-MDT0000 complete [ 1277.925122] LustreError: 39973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.934666] LustreError: 39973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1278.214166] LustreError: 39954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752764 with bad export cookie 1168417040894382990 [ 1278.224185] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.626081] Lustre: server umount lustre-MDT0001 complete [ 1294.462514] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 1297.591793] Key type lgssc unregistered [ 1297.975568] LNet: 43686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.985427] LNetError: 43686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.063896] LNet: Removed LNI 192.168.201.154@tcp [ 1300.113139] Key type .llcrypt unregistered [ 1300.119612] Key type ._llcrypt unregistered [ 1330.271459] Key type ._llcrypt registered [ 1330.274245] Key type .llcrypt registered [ 1330.399317] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 1331.543289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.766180] alg: No test for adler32 (adler32-zlib) [ 1332.890507] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1333.263640] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 1335.010503] Key type lgssc registered [ 1335.915429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.929372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.946996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.322343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.191164] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1353.763855] LustreError: 45098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.784661] LustreError: 45098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1357.724569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.230930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1363.752398] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1371.410961] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1377.944407] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1385.126395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.573663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1389.683365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1392.060391] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1399.204572] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.739868] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.54@tcp (stopping) [ 1413.613447] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1413.625979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1413.626857] Lustre: Skipped 1 previous similar message [ 1413.647090] Lustre: Skipped 2 previous similar messages [ 1416.627391] Lustre: server umount lustre-MDT0000 complete [ 1417.833807] LustreError: 47590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.049314] LustreError: 45080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752907 with bad export cookie 13992671798791911849 [ 1421.053993] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1421.067146] LustreError: 45080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1422.957583] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.983095] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1423.005406] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.54@tcp (stopping) [ 1423.869025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.887202] Lustre: Skipped 1 previous similar message [ 1426.853634] LustreError: 47589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.854822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1426.888116] LustreError: 47589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1426.931971] Lustre: Skipped 1 previous similar message [ 1427.661833] Lustre: server umount lustre-MDT0001 complete [ 1442.845266] Lustre: server umount lustre-OST0000 complete [ 1464.583385] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 1467.613089] Key type lgssc unregistered [ 1467.938842] LNet: 48965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.942304] LNetError: 48965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.954617] LNet: Removed LNI 192.168.201.154@tcp [ 1468.859178] Key type .llcrypt unregistered [ 1468.863719] Key type ._llcrypt unregistered [ 1494.752045] Key type ._llcrypt registered [ 1494.761033] Key type .llcrypt registered [ 1494.915612] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 1496.139581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.339231] alg: No test for adler32 (adler32-zlib) [ 1497.474263] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1497.750234] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 1499.418300] Key type lgssc registered [ 1501.478184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1511.028878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1511.064641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.687575] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1531.808488] LustreError: 50354:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 1558.499254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.882473] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1573.175576] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1740.657143] Lustre: server umount lustre-OST0000 complete [ 1756.773488] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 1759.452095] Key type lgssc unregistered [ 1759.680901] LNet: 52034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.686389] LNetError: 52034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.701546] LNet: Removed LNI 192.168.201.154@tcp [ 1760.468456] Key type .llcrypt unregistered [ 1760.470378] Key type ._llcrypt unregistered [ 1782.841687] Key type ._llcrypt registered [ 1782.843104] Key type .llcrypt registered [ 1782.928075] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 1783.894107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.996090] alg: No test for adler32 (adler32-zlib) [ 1785.144442] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1785.331078] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 1786.967688] Key type lgssc registered [ 1788.041857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1794.947771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1794.964769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.455574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.738556] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1805.806232] LustreError: 53448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.821855] LustreError: 53448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1808.458731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.930918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1813.189792] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1819.821542] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1824.727337] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1830.258661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1830.655097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.520775] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1840.975957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1843.259607] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1846.251080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.252392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1846.254514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.259335] Lustre: Skipped 1 previous similar message [ 1847.364193] Lustre: server umount lustre-OST0000 complete [ 1854.946070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1854.957736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.976725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.983526] Lustre: Skipped 1 previous similar message [ 1856.480900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.483798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1858.133556] Lustre: server umount lustre-MDT0000 complete [ 1861.607141] LustreError: 53443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.639643] LustreError: 53443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1861.957160] LustreError: 53429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753348 with bad export cookie 416527835818081703 [ 1861.973887] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1861.977617] LustreError: 53429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1862.232280] Lustre: server umount lustre-MDT0001 complete [ 1871.630931] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 1875.067349] Key type lgssc unregistered [ 1875.328769] LNet: 56988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.340677] LNetError: 56988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.360891] LNet: Removed LNI 192.168.201.154@tcp [ 1876.246945] Key type .llcrypt unregistered [ 1876.248741] Key type ._llcrypt unregistered [ 1898.823221] Key type ._llcrypt registered [ 1898.826112] Key type .llcrypt registered [ 1898.931833] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 1899.887459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.998961] alg: No test for adler32 (adler32-zlib) [ 1901.014577] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1901.236906] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 1902.919177] Key type lgssc registered [ 1903.975379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1910.697877] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1910.728179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1926.303720] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1931.423667] LustreError: 58373:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 1958.112843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.120032] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1970.959587] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1977.553173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.941316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.029062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1982.801362] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 1982.947711] LustreError: 58398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.972673] LustreError: 58398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1988.078383] LustreError: 58399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1990.804745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.171639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1995.460337] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2002.331253] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2007.686541] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2011.618317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.631135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.660871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.676305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2013.682807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2013.697682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.058297] Lustre: server umount lustre-OST0000 complete [ 2037.216298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2037.221877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.229756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.234875] Lustre: Skipped 1 previous similar message [ 2039.264343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.266398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.367568] Lustre: server umount lustre-MDT0000 complete [ 2044.391770] LustreError: 59414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.868720] LustreError: 58380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753533 with bad export cookie 7144130104890706604 [ 2046.891413] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.898047] LustreError: 58380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2061.279733] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2061.612875] Lustre: server umount lustre-MDT0001 complete [ 2072.703372] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2075.739758] Key type lgssc unregistered [ 2076.070923] LNet: 62118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.081080] LNetError: 62118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.102739] LNet: Removed LNI 192.168.201.154@tcp [ 2077.029704] Key type .llcrypt unregistered [ 2077.031953] Key type ._llcrypt unregistered [ 2101.972729] Key type ._llcrypt registered [ 2101.974315] Key type .llcrypt registered [ 2102.058556] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 2103.411908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2103.438403] alg: No test for adler32 (adler32-zlib) [ 2104.594266] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2104.770717] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 2106.407283] Key type lgssc registered [ 2107.464113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2114.921170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2114.954228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.594360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.559733] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2125.801218] LustreError: 63534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.845842] LustreError: 63534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2130.915226] LustreError: 63533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.407365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.818192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.790386] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2145.014937] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.519195] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2157.381526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2157.483279] LustreError: 65354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.696796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.834380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2164.183631] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2172.153928] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2196.699851] Lustre: server umount lustre-OST0000 complete [ 2202.081342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.087880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.105663] Lustre: Skipped 1 previous similar message [ 2202.114175] Lustre: Skipped 1 previous similar message [ 2207.206589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.227780] Lustre: Skipped 1 previous similar message [ 2212.323568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2212.329160] Lustre: Skipped 1 previous similar message [ 2213.344236] Lustre: 62847:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753683/real 1786753683] req@0000000097182bd9 x1873546933468288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786753699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2213.346326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.374220] Lustre: 62847:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2215.393481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2215.639758] Lustre: server umount lustre-MDT0000 complete [ 2218.464478] LustreError: 63534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.483149] LustreError: 63534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2218.529943] Lustre: 62844:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753688/real 1786753688] req@00000000ea0b1acd x1873546933470336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786753704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2220.200360] LustreError: 63514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753706 with bad export cookie 11984494488497976247 [ 2220.208775] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.213573] LustreError: 63514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2220.525791] Lustre: server umount lustre-MDT0001 complete [ 2230.108682] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2233.925320] Key type lgssc unregistered [ 2234.315796] LNet: 67098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.321803] LNetError: 67098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.334228] LNet: Removed LNI 192.168.201.154@tcp [ 2235.351542] Key type .llcrypt unregistered [ 2235.354537] Key type ._llcrypt unregistered [ 2264.270671] Key type ._llcrypt registered [ 2264.273255] Key type .llcrypt registered [ 2264.410399] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 2265.429326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2265.514456] alg: No test for adler32 (adler32-zlib) [ 2266.665680] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2266.989801] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 2268.743994] Key type lgssc registered [ 2270.350406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2283.780896] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:29:27 (1786753767) [ 2292.079709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2292.112967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.699124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.580270] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2303.985102] LustreError: 68808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.004279] LustreError: 68808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2305.002547] LustreError: 68807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.720704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.463911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2313.861334] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2322.947828] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.145147] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2335.168910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.593058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.703811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2341.824607] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2350.245719] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2358.946534] Lustre: Failing over lustre-MDT0000 [ 2359.265632] Lustre: server umount lustre-MDT0000 complete [ 2359.796634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2359.803547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.817436] LustreError: 68807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.019405] LustreError: 68803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.367200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.383397] Lustre: Skipped 1 previous similar message [ 2367.088861] LustreError: 68804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.105754] LustreError: 68804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2368.572053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.764260] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2368.940794] Lustre: *** cfs_fail_loc=135, val=0*** [ 2369.015276] LustreError: 71894:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2369.021688] LustreError: 71894:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.154@tcp: cfg command failed: rc = -2 [ 2369.032654] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2369.032654] [ 2369.042165] LustreError: MGC192.168.201.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2369.056132] LustreError: 71883:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2369.064338] LustreError: 71883:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2369.072749] LustreError: 71883:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2369.079684] LustreError: 71883:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2369.097931] Lustre: server umount lustre-MDT0000 complete [ 2369.102243] LustreError: 71883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2375.806678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.041176] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2376.052752] LustreError: 72239:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.154@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000005a2cb10b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2376.067042] LustreError: 68799:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2376.358630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.430494] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2377.326073] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2381.256279] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2381.805970] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2381.861917] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2381.937171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2388.962344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2388.968707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.984538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.041703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2392.052527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.079704] Lustre: Skipped 1 previous similar message [ 2397.168685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.181714] Lustre: Skipped 1 previous similar message [ 2402.278122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.299607] Lustre: Skipped 1 previous similar message [ 2403.296173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2403.441558] Lustre: server umount lustre-OST0000 complete [ 2412.513385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.515122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.525974] Lustre: Skipped 1 previous similar message [ 2412.549829] Lustre: Skipped 1 previous similar message [ 2422.239216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2422.522285] Lustre: server umount lustre-MDT0000 complete [ 2422.752067] LustreError: 68808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.762402] LustreError: 68808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2425.972954] LustreError: 68787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753912 with bad export cookie 5244321630817736358 [ 2425.984374] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.990173] LustreError: 68787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2426.248967] Lustre: server umount lustre-MDT0001 complete [ 2437.070209] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2439.367478] Key type lgssc unregistered [ 2439.730969] LNet: 73677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.743296] LNetError: 73677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.760822] LNet: Removed LNI 192.168.201.154@tcp [ 2440.800040] Key type .llcrypt unregistered [ 2440.804118] Key type ._llcrypt unregistered [ 2465.210508] Key type ._llcrypt registered [ 2465.213974] Key type .llcrypt registered [ 2465.410903] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 2466.810043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2466.897910] alg: No test for adler32 (adler32-zlib) [ 2468.223426] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2468.595886] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 2470.271276] Key type lgssc registered [ 2471.669646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2480.634628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2480.647171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.187250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.328292] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2492.391340] LustreError: 75101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.420045] LustreError: 75101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2496.090933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.358174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.212868] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2507.970590] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2514.530563] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2521.289833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2521.619980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.731233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2528.020572] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2535.301387] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.342780] Lustre: Failing over lustre-MDT0000 [ 2545.692757] Lustre: server umount lustre-MDT0000 complete [ 2546.155208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.156925] LustreError: 75100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.164348] Lustre: Skipped 1 previous similar message [ 2546.207249] LustreError: 75100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2548.376901] LustreError: 76945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.271794] LustreError: 77702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.309232] LustreError: 77702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2555.228914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.310566] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.404311] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2555.584478] LustreError: 78192:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2555.589383] LustreError: 78192:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.154@tcp: cfg command failed: rc = -9 [ 2555.594017] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2555.594017] [ 2555.602156] LustreError: MGC192.168.201.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2555.629540] LustreError: 78180:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2555.641201] LustreError: 78180:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2555.647520] LustreError: 78180:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2555.660777] LustreError: 78180:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2555.722339] Lustre: server umount lustre-MDT0000 complete [ 2555.725637] LustreError: 78180:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2556.384401] LustreError: 75830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.406560] LustreError: 75830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2563.122209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.246239] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2563.638276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.775338] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2567.775325] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2568.681305] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2568.709056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2568.861892] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2568.941621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2576.869192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2576.889805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.916201] Lustre: Skipped 1 previous similar message [ 2576.924599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.918234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2578.933602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.055758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2584.074917] Lustre: Skipped 2 previous similar messages [ 2589.154525] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2589.165375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2589.176447] Lustre: Skipped 1 previous similar message [ 2589.580630] Lustre: server umount lustre-OST0000 complete [ 2594.273418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.299419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.314497] Lustre: Skipped 1 previous similar message [ 2600.430726] Lustre: server umount lustre-MDT0000 complete [ 2604.502512] LustreError: 75837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754090 with bad export cookie 11123873811410680414 [ 2604.511903] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.516057] LustreError: 75837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2605.061063] Lustre: server umount lustre-MDT0001 complete [ 2618.040737] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2620.994655] Key type lgssc unregistered [ 2621.369426] LNet: 79979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.378371] LNetError: 79979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.404662] LNet: Removed LNI 192.168.201.154@tcp [ 2622.323726] Key type .llcrypt unregistered [ 2622.330467] Key type ._llcrypt unregistered [ 2634.714498] Key type ._llcrypt registered [ 2634.716715] Key type .llcrypt registered [ 2634.818863] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2635.291409] Key type .llcrypt unregistered [ 2635.293076] Key type ._llcrypt unregistered [ 2659.239244] Key type ._llcrypt registered [ 2659.252772] Key type .llcrypt registered [ 2659.446597] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 2660.750510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2660.854595] alg: No test for adler32 (adler32-zlib) [ 2661.924212] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2662.246611] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 2663.903238] Key type lgssc registered [ 2665.409820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2673.875439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2673.905567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.404953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.820844] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2684.903989] LustreError: 81885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.916677] LustreError: 81885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2689.308723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.938891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.200574] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2705.198045] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2711.720973] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2719.772112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.338462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.623509] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2735.074126] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2744.951360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2760.672097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2760.675636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.677800] LustreError: Skipped 1 previous similar message [ 2760.680877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.696844] Lustre: Skipped 1 previous similar message [ 2762.176096] Lustre: server umount lustre-OST0000 complete [ 2766.817864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.825148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.837232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.844158] Lustre: Skipped 1 previous similar message [ 2770.913312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.922730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.744234] Lustre: server umount lustre-MDT0000 complete [ 2776.032362] LustreError: 81886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.064173] LustreError: 81886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2776.937310] LustreError: 84483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754263 with bad export cookie 13102693138684533398 [ 2776.939725] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.945635] LustreError: 84483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2777.273345] Lustre: server umount lustre-MDT0001 complete [ 2789.112901] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2791.448308] Key type lgssc unregistered [ 2791.701752] LNet: 85440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.706752] LNetError: 85440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.747769] LNet: Removed LNI 192.168.201.154@tcp [ 2793.675874] Key type .llcrypt unregistered [ 2793.677731] Key type ._llcrypt unregistered [ 2820.760282] Key type ._llcrypt registered [ 2820.762769] Key type .llcrypt registered [ 2820.932344] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 2822.013770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2822.060177] alg: No test for adler32 (adler32-zlib) [ 2823.216910] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2823.420005] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 2825.095156] Key type lgssc registered [ 2826.678952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2835.429397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2835.473476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.008616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.649251] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2847.204390] LustreError: 86853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.220247] LustreError: 86853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2850.594838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.874465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2855.669916] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2864.215312] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2870.676696] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2877.863162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2878.407613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.504863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2885.890614] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2894.993079] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2900.963085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.990160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.137180] Lustre: server umount lustre-OST0000 complete [ 2907.106081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2907.120976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.145756] Lustre: Skipped 1 previous similar message [ 2907.159255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.172990] Lustre: Skipped 1 previous similar message [ 2911.212568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.213963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.078151] Lustre: server umount lustre-MDT0000 complete [ 2916.325318] LustreError: 86854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.345655] LustreError: 86854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2916.461893] LustreError: 86835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754402 with bad export cookie 11078844979179054549 [ 2916.479049] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.485602] LustreError: 86835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2916.875163] Lustre: server umount lustre-MDT0001 complete [ 2928.783710] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 2931.835589] Key type lgssc unregistered [ 2932.158592] LNet: 90387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2932.162163] LNetError: 90387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2932.173882] LNet: Removed LNI 192.168.201.154@tcp [ 2933.013408] Key type .llcrypt unregistered [ 2933.015522] Key type ._llcrypt unregistered [ 2956.694899] Key type ._llcrypt registered [ 2956.698555] Key type .llcrypt registered [ 2956.802602] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 2958.015632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2958.146147] alg: No test for adler32 (adler32-zlib) [ 2959.140614] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2959.357332] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 2961.023718] Key type lgssc registered [ 2962.021339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.265542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.278582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.716136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.678372] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2980.837403] LustreError: 91798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.863497] LustreError: 91798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2983.929933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.364740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.816573] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 2996.173482] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3002.338282] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3009.206267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.547033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.473268] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3016.697356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3025.438865] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3041.774473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3041.781228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.800422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3044.326551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3044.328632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3044.333123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.363348] Lustre: Skipped 1 previous similar message [ 3047.388454] Lustre: server umount lustre-OST0000 complete [ 3054.566333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.578391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.905542] Lustre: server umount lustre-MDT0000 complete [ 3059.686945] LustreError: 92526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.720174] LustreError: 92526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3062.010442] LustreError: 91779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754548 with bad export cookie 12052386841703295855 [ 3062.012115] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.032955] LustreError: 91779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3062.352213] Lustre: server umount lustre-MDT0001 complete [ 3073.905858] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 3076.768941] Key type lgssc unregistered [ 3077.099240] LNet: 95358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3077.116995] LNetError: 95358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3078.186552] LNet: Removed LNI 192.168.201.154@tcp [ 3079.297890] Key type .llcrypt unregistered [ 3079.300105] Key type ._llcrypt unregistered [ 3104.104699] Key type ._llcrypt registered [ 3104.107403] Key type .llcrypt registered [ 3104.233248] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 3105.415588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3105.510073] alg: No test for adler32 (adler32-zlib) [ 3106.546247] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3106.812983] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 3108.492473] Key type lgssc registered [ 3110.178257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3119.109800] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3119.142303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.687444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3139.807478] LustreError: 96747:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 3166.561902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.463700] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3181.770494] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3188.374398] Lustre: server umount lustre-OST0000 complete [ 3196.998149] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:44:41 (1786754681) [ 3203.087074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.843613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.189543] LustreError: 98391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.207951] LustreError: 98391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3209.358346] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3214.308550] LustreError: 98390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.428233] LustreError: 98391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.869673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.237970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.509293] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3233.210776] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3238.059959] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3243.896143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3244.260824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3250.429361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3250.509414] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3257.608187] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3271.143037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.144830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.160290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.192620] Lustre: Skipped 1 previous similar message [ 3274.500028] Lustre: server umount lustre-OST0000 complete [ 3281.388665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.390499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.404599] Lustre: Skipped 1 previous similar message [ 3281.429198] Lustre: Skipped 2 previous similar messages [ 3284.461940] Lustre: server umount lustre-MDT0000 complete [ 3286.502671] LustreError: 100232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.520131] LustreError: 100232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3288.108463] LustreError: 98370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754774 with bad export cookie 12234680915825017168 [ 3288.122448] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.123146] LustreError: 98370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3288.376151] Lustre: server umount lustre-MDT0001 complete [ 3301.716793] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 3305.744855] Key type lgssc unregistered [ 3306.185622] LNet: 102090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.213677] LNetError: 102090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.255794] LNet: Removed LNI 192.168.201.154@tcp [ 3308.118338] Key type .llcrypt unregistered [ 3308.120926] Key type ._llcrypt unregistered [ 3349.163419] Key type ._llcrypt registered [ 3349.165776] Key type .llcrypt registered [ 3349.269759] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 3350.555446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3350.590886] alg: No test for adler32 (adler32-zlib) [ 3351.771648] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3352.068615] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 3353.799390] Key type lgssc registered [ 3354.652791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3361.675345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3361.706632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.296780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.290793] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3373.556937] LustreError: 104287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.593227] LustreError: 104287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3377.451786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.109699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3382.370442] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3389.890149] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3395.846548] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3401.502156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.793264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.303674] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3407.911078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3415.228422] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3428.323824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.326417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.338489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.352870] LustreError: Skipped 1 previous similar message [ 3430.233615] Lustre: server umount lustre-OST0000 complete [ 3434.464730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3434.473496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.491052] Lustre: Skipped 1 previous similar message [ 3434.504631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3434.520043] Lustre: Skipped 1 previous similar message [ 3438.560826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.561572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.605819] Lustre: Skipped 1 previous similar message [ 3440.403568] Lustre: server umount lustre-MDT0000 complete [ 3443.685961] LustreError: 104282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.700363] LustreError: 104282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3443.747110] LustreError: 104268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754930 with bad export cookie 7266988238129651 [ 3443.755619] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3444.008893] Lustre: server umount lustre-MDT0001 complete [ 3454.535616] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 3457.202651] Key type lgssc unregistered [ 3457.568779] LNet: 107820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3457.575611] LNetError: 107820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3458.602156] LNet: Removed LNI 192.168.201.154@tcp [ 3459.471378] Key type .llcrypt unregistered [ 3459.477679] Key type ._llcrypt unregistered [ 3479.169878] Key type ._llcrypt registered [ 3479.173538] Key type .llcrypt registered [ 3479.300662] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 3480.444416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3480.469838] alg: No test for adler32 (adler32-zlib) [ 3481.914721] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3482.339641] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 3484.159699] Key type lgssc registered [ 3486.166693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3496.779426] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3496.822642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.479701] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3517.599460] LustreError: 109062:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 3544.287245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.585778] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3557.069778] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3563.608257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.804466] LustreError: MGC192.168.201.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3563.814243] LustreError: 110086:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3563.819465] LustreError: 110086:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3563.827376] LustreError: 110086:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3563.837134] LustreError: 109070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755050 with bad export cookie 18440228913780803508 [ 3563.854847] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.943647] Lustre: server umount lustre-MDT0000 complete [ 3563.952424] LustreError: 110086:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3589.759420] Lustre: server umount lustre-OST0000 complete [ 3600.070073] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_hostid [ 3610.239784] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 3641.052275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.315376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3641.337908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3641.422629] Lustre: lustre-MDT0000: new disk, initializing [ 3641.505878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.525878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3645.560428] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3655.492317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.617985] Lustre: 112879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3655.675621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3655.679905] Lustre: Skipped 1 previous similar message [ 3655.754635] Lustre: lustre-MDT0001: new disk, initializing [ 3655.865838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3655.912138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3655.921173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3660.460049] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3669.230354] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3674.920113] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3681.449454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3681.765345] Lustre: lustre-OST0000: new disk, initializing [ 3681.768140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3681.773216] Lustre: 114848:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3681.845798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.918652] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3691.546926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3691.558393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3691.646358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.142971] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3700.509950] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3700.731860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3705.439155] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3705.638519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3711.989697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.990777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.001566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.014531] Lustre: Skipped 1 previous similar message [ 3713.737142] Lustre: server umount lustre-OST0000 complete [ 3717.600698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3717.607391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.621264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.626896] Lustre: Skipped 1 previous similar message [ 3722.210381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.215725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3723.888722] Lustre: server umount lustre-MDT0000 complete [ 3727.253922] LustreError: 114847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755213 with bad export cookie 18440228913780803613 [ 3727.263233] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3727.267554] LustreError: 114847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3727.508441] Lustre: server umount lustre-MDT0001 complete [ 3735.375264] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:53:40 (1786755220) [ 3738.624424] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3754.812333] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_hostid [ 3764.557313] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 3798.694498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.932865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3798.952503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3799.024358] Lustre: lustre-MDT0000: new disk, initializing [ 3799.097691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.112866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3803.736880] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3813.301269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.485469] Lustre: 119748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3813.540263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3813.543731] Lustre: Skipped 1 previous similar message [ 3813.601062] Lustre: lustre-MDT0001: new disk, initializing [ 3813.697941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3813.749748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3813.765616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3818.518750] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3827.151331] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3832.013594] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3838.472336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.827972] Lustre: lustre-OST0000: new disk, initializing [ 3838.858287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3838.862818] Lustre: 121714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3838.924345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.077844] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3848.794030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3848.814129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3848.993849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3854.062260] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3858.591522] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3858.870653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3862.940354] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3863.255357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3869.665154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3869.667330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.686112] LustreError: Skipped 1 previous similar message [ 3869.690592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3869.709081] Lustre: Skipped 1 previous similar message [ 3869.740291] Lustre: Skipped 2 previous similar messages [ 3871.958259] Lustre: server umount lustre-OST0000 complete [ 3879.905227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.920171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3879.932257] Lustre: Skipped 1 previous similar message [ 3881.773755] Lustre: server umount lustre-MDT0000 complete [ 3885.050963] LustreError: 119741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755371 with bad export cookie 18440228913780804495 [ 3885.052417] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3885.057618] LustreError: 119741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3885.401874] Lustre: server umount lustre-MDT0001 complete [ 3892.537270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.829657] LustreError: 123932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.849401] LustreError: 123932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3892.909797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.807163] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3898.336742] LustreError: 123933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.469822] LustreError: 123932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.690092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.403420] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3917.423862] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3922.436879] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3929.224833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3935.668188] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 3942.060901] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3948.198948] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3953.122244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.131961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.131970] Lustre: Skipped 1 previous similar message [ 3953.133165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.696465] Lustre: server umount lustre-OST0000 complete [ 3962.338729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3968.034231] Lustre: server umount lustre-MDT0000 complete [ 3968.481654] LustreError: 124941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.512038] LustreError: 124941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3971.858392] LustreError: 123912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755458 with bad export cookie 18440228913780805377 [ 3971.877704] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3971.878505] LustreError: 123912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3972.179769] Lustre: server umount lustre-MDT0001 complete [ 3980.897445] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 3983.685142] Key type lgssc unregistered [ 3983.927708] LNet: 127565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3983.932141] LNetError: 127565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3983.949146] LNet: Removed LNI 192.168.201.154@tcp [ 3984.852798] Key type .llcrypt unregistered [ 3984.855242] Key type ._llcrypt unregistered [ 4008.762438] Key type ._llcrypt registered [ 4008.766105] Key type .llcrypt registered [ 4008.905332] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_hostid [ 4024.313901] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 4025.354958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4025.497155] alg: No test for adler32 (adler32-zlib) [ 4026.472949] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 4026.712671] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 4028.401887] Key type lgssc registered [ 4029.393428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4067.290149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4067.330254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.590717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4068.631459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4068.728267] Lustre: lustre-MDT0000: new disk, initializing [ 4068.796233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.817600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4073.453750] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4084.431894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.547681] Lustre: 130723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4084.661750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4084.668728] Lustre: Skipped 1 previous similar message [ 4084.791905] Lustre: lustre-MDT0001: new disk, initializing [ 4084.939226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4085.007760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4085.021179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4088.668958] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4099.105252] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4105.013222] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4112.497782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.788616] Lustre: lustre-OST0000: new disk, initializing [ 4112.792405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4112.797482] Lustre: 132691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4112.895352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.110252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4118.118944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4118.226564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4120.215184] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4128.740921] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4133.183427] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4133.440346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4137.802992] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4137.986741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4144.096292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4144.102221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.107392] LustreError: Skipped 1 previous similar message [ 4144.109517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4144.159860] Lustre: Skipped 1 previous similar message [ 4145.896108] Lustre: server umount lustre-OST0000 complete [ 4151.783608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4151.789639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.798807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.801608] Lustre: Skipped 1 previous similar message [ 4154.349674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.354521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.984925] Lustre: server umount lustre-MDT0000 complete [ 4162.327038] LustreError: 131518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755648 with bad export cookie 14817335060164938322 [ 4162.342599] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4162.355463] LustreError: 131518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4162.861530] Lustre: server umount lustre-MDT0001 complete [ 4172.701994] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:00:57 (1786755657) [ 4179.734370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.354748] LustreError: 135114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.377794] LustreError: 135114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4180.521293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.568795] LustreError: 135115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.812915] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4189.666447] LustreError: 135114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.789405] LustreError: 135115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.458722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4195.800375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4200.750247] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4208.172840] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4213.278194] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4216.293984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4216.295664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.312099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.331429] Lustre: Skipped 1 previous similar message [ 4216.346440] Lustre: Skipped 2 previous similar messages [ 4221.415713] LustreError: 135115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.424238] LustreError: 135115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4221.571575] Lustre: server umount lustre-MDT0000 complete [ 4225.163938] LustreError: 135096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755711 with bad export cookie 14817335060164939204 [ 4225.169918] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4225.176992] LustreError: 135096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4225.680215] Lustre: server umount lustre-MDT0001 complete [ 4234.401698] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:01:59 (1786755719) [ 4242.798308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4258.336133] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4263.455625] LustreError: 137531:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 4289.185320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.581092] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4301.429312] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4303.468238] Lustre: server umount lustre-OST0000 complete [ 4311.651900] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:03:16 (1786755796) [ 4317.613272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.949205] LustreError: 138976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.964378] LustreError: 138976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4318.015813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.374657] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4331.041779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.590980] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4343.213891] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4348.104993] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4354.737474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.129966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.137495] Lustre: Skipped 1 previous similar message [ 4362.909985] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4370.040150] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4377.568856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4377.584389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.603913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.583361] Lustre: server umount lustre-MDT0000 complete [ 4384.225291] LustreError: 138977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.246766] LustreError: 138977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4387.522049] LustreError: 138957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755873 with bad export cookie 14817335060164939862 [ 4387.531593] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4387.534568] LustreError: 138957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4387.827744] Lustre: server umount lustre-MDT0001 complete [ 4401.447573] Lustre: server umount lustre-OST0000 complete [ 4409.155664] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:04:53 (1786755893) [ 4414.999496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.570714] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4420.580417] LustreError: 142629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.580429] LustreError: 142629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4429.114148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.455259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4429.459657] Lustre: Skipped 1 previous similar message [ 4434.268579] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4442.336244] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4447.417223] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4454.596977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.038951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4461.361302] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4468.947439] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4474.746492] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4474.885529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4479.937370] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4480.153129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4484.588299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4484.591513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.606207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4484.620420] Lustre: Skipped 3 previous similar messages [ 4484.625606] Lustre: Skipped 3 previous similar messages [ 4496.351135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4496.602404] Lustre: server umount lustre-OST0000 complete [ 4505.058499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4505.070839] Lustre: Skipped 8 previous similar messages [ 4514.271390] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4514.585804] Lustre: server umount lustre-MDT0000 complete [ 4515.297611] LustreError: 142626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.312425] LustreError: 142626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4518.175816] LustreError: 142609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756004 with bad export cookie 14817335060164940940 [ 4518.193470] LustreError: 142609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4518.210523] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.513254] Lustre: server umount lustre-MDT0001 complete [ 4526.196493] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:06:51 (1786756011) [ 4532.654553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4548.191710] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4553.311714] LustreError: 146787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 4579.039353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4579.052204] Lustre: Skipped 1 previous similar message [ 4587.662316] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4597.645960] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4607.410151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.194417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4612.962900] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4621.303364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.207931] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4633.771764] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4638.832089] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4643.387850] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4643.857635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4649.037644] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4649.212865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4652.515573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4652.517983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.541302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4652.554064] Lustre: Skipped 3 previous similar messages [ 4652.571645] Lustre: Skipped 4 previous similar messages [ 4657.495252] Lustre: server umount lustre-MDT0000 complete [ 4657.633933] LustreError: 148561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.666730] LustreError: 148561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4662.154303] LustreError: 146796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756148 with bad export cookie 14817335060164941962 [ 4662.161313] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4662.188065] LustreError: 146796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4662.561897] Lustre: server umount lustre-MDT0001 complete [ 4677.360738] Lustre: server umount lustre-OST0000 complete [ 4686.055430] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:09:31 (1786756171) [ 4691.926662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4707.551626] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4712.671518] LustreError: 150985:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 4745.946312] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4752.925709] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4759.474115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.646552] LustreError: 128943:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000062229138 x1873548948955648/t0(0) o250->MGC192.168.201.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4760.171388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4764.196673] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4771.994969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.398316] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4783.571198] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4789.147673] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4795.148625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4795.363966] Lustre: lustre-OST0001: new disk, initializing [ 4795.367417] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4795.372566] Lustre: 153855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4801.704965] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4802.627832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4802.643317] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4802.790765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4810.371678] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4814.608815] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4814.807909] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4818.998304] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4819.304890] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4821.472741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4821.480610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.487569] Lustre: Skipped 1 previous similar message [ 4821.502092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4835.297929] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4835.446366] Lustre: server umount lustre-OST0000 complete [ 4845.574955] Lustre: server umount lustre-OST0001 complete [ 4854.987040] Lustre: server umount lustre-MDT0000 complete [ 4858.404691] LustreError: 150992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756344 with bad export cookie 14817335060164942914 [ 4858.407803] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4858.415125] LustreError: 150992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4858.930806] Lustre: server umount lustre-MDT0001 complete [ 4880.667396] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:12:46 (1786756366) [ 4882.289291] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4884.368433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4886.446505] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:12:50 (1786756370) [ 4894.817321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.021276] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4895.034930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4895.198560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.208513] Lustre: Skipped 6 previous similar messages [ 4900.564866] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4909.591408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.651975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4909.682750] Lustre: 157419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4913.270518] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4919.985802] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4925.917550] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4932.753050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.979216] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4939.638679] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 4941.547580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4946.403477] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4951.222670] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4951.397876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4956.148625] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4956.332954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4962.284883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4962.285918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.294054] LustreError: Skipped 5 previous similar messages [ 4962.296901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4962.296909] Lustre: Skipped 12 previous similar messages [ 4962.345774] Lustre: Skipped 6 previous similar messages [ 4964.645894] Lustre: server umount lustre-OST0000 complete [ 4987.473935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4994.857068] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5002.055433] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5002.164837] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.194279] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5002.205703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.460790] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5008.801786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5013.128091] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5013.300954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5019.536502] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f104a809000.ost_server_uuid 50 [ 5021.211919] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f104a809000.ost_server_uuid in FULL state after 0 sec [ 5021.568902] LustreError: 158282:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.54@tcp) returned error from blocking AST (req@00000000cdd489ae x1873548949062912 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000022563c17/0xcda1bfbb4b12a59c lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.54@tcp remote: 0xcbac5792c0ef823b expref: 5 pid: 157426 timeout: 5121 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5021.605472] LustreError: lustre-MDT0000: A client on nid 192.168.201.54@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5021.614537] LustreError: 157414:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.54@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000022563c17/0xcda1bfbb4b12a59c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.54@tcp remote: 0xcbac5792c0ef823b expref: 6 pid: 157426 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5026.798523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5032.371598] Lustre: server umount lustre-OST0000 complete [ 5050.335295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5050.575695] Lustre: server umount lustre-MDT0000 complete [ 5053.407991] LustreError: 158787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.416699] LustreError: 158787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5053.784678] LustreError: 157409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756540 with bad export cookie 14817335060164944097 [ 5053.788629] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5053.803228] LustreError: 157409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5068.255438] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5068.637365] Lustre: server umount lustre-MDT0001 complete [ 5080.511933] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5082.657952] Key type lgssc unregistered [ 5082.879691] LNet: 163540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5082.883948] LNetError: 163540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5083.945711] LNet: Removed LNI 192.168.201.154@tcp [ 5084.688128] Key type .llcrypt unregistered [ 5084.689851] Key type ._llcrypt unregistered [ 5108.060989] Key type ._llcrypt registered [ 5108.063593] Key type .llcrypt registered [ 5108.152249] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 5109.502563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5109.525826] alg: No test for adler32 (adler32-zlib) [ 5110.753696] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5111.070424] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 5112.769541] Key type lgssc registered [ 5113.824266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5120.767340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5120.788783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.128202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.478419] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5132.258922] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.292155] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5135.538836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.078262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.671227] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5148.688535] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5155.158536] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5162.731399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5162.983486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.094168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5168.891819] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5177.519599] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5187.201279] Lustre: Failing over lustre-MDT0000 [ 5187.446937] Lustre: server umount lustre-MDT0000 complete [ 5191.657460] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.658723] LustreError: 164956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.676745] Lustre: Skipped 2 previous similar messages [ 5191.727901] LustreError: 164956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5196.769077] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.784381] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5200.267372] LustreError: 164935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.54@tcp arrived at 1786756686 with bad export cookie 13412265680376113660 [ 5200.302833] LustreError: 164935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5201.892097] LustreError: 165683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.930062] LustreError: 165683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5207.009412] LustreError: 164956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.086354] LustreError: 164956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5208.095115] Lustre: 164267:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756678/real 1786756678] req@000000004842b733 x1873550085484288/t0(0) o400->MGC192.168.201.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786756694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5208.115642] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5208.751920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.272849] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.274446] LustreError: 164263:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009b01f6f7 x1873550085490176/t0(0) o250->MGC192.168.201.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5218.290654] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5218.619392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.690272] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5223.391988] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5223.913579] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5223.927183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5225.690471] Lustre: lustre-MDT0000: Denying connection for new client 7bb2c361-20ed-429e-8808-6b8a1badb74c (at 192.168.201.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5230.728424] Lustre: lustre-MDT0000: Denying connection for new client 7bb2c361-20ed-429e-8808-6b8a1badb74c (at 192.168.201.54@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5234.155809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.169997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.175045] Lustre: Skipped 1 previous similar message [ 5234.192686] Lustre: Skipped 1 previous similar message [ 5239.268272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.283049] Lustre: Skipped 1 previous similar message [ 5240.315181] Lustre: server umount lustre-OST0000 complete [ 5244.206949] LustreError: 168686:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5244.213513] Lustre: 168035:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5244.220955] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5244.237731] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5244.254528] LustreError: 168035:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d2738a64 x1873550085503616/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5244.275324] LustreError: 168035:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5244.282069] LustreError: 168035:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5244.299559] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5244.385622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.413118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.643566] Lustre: server umount lustre-MDT0000 complete [ 5248.635846] LustreError: 164936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756734 with bad export cookie 13412265680376113765 [ 5248.640963] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5248.643121] LustreError: 164955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.644201] LustreError: 164936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5251.018702] Lustre: server umount lustre-MDT0001 complete [ 5261.446592] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5264.322242] Key type lgssc unregistered [ 5264.769708] LNet: 169442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5264.779157] LNetError: 169442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5264.810863] LNet: Removed LNI 192.168.201.154@tcp [ 5265.717432] Key type .llcrypt unregistered [ 5265.722591] Key type ._llcrypt unregistered [ 5291.414524] Key type ._llcrypt registered [ 5291.416858] Key type .llcrypt registered [ 5291.509256] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 5292.974504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5293.060840] alg: No test for adler32 (adler32-zlib) [ 5294.416774] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5294.892710] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 5296.664350] Key type lgssc registered [ 5298.513821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5309.940382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5309.971055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.362383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.642445] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5320.684195] LustreError: 170857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.718552] LustreError: 170857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5324.780524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.228237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5330.592441] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5339.175852] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5345.504447] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5352.609344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.052698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.159220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5360.943837] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5369.616340] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5375.969372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.983177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.992959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5376.014874] Lustre: Skipped 1 previous similar message [ 5381.090234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.111946] Lustre: Skipped 1 previous similar message [ 5386.222653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5386.233944] Lustre: Skipped 1 previous similar message [ 5390.303271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5390.549631] Lustre: server umount lustre-OST0000 complete [ 5396.451419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.460089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5396.462023] Lustre: Skipped 1 previous similar message [ 5400.845269] Lustre: server umount lustre-MDT0000 complete [ 5401.569939] LustreError: 170853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.580311] LustreError: 170853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5404.140135] LustreError: 170837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756890 with bad export cookie 1671757433257064594 [ 5404.144567] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.149912] LustreError: 170837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5404.831092] Lustre: server umount lustre-MDT0001 complete [ 5417.438677] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5421.123927] Key type lgssc unregistered [ 5421.454408] LNet: 174393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5421.458054] LNetError: 174393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5421.522781] LNet: Removed LNI 192.168.201.154@tcp [ 5422.771837] Key type .llcrypt unregistered [ 5422.775301] Key type ._llcrypt unregistered [ 5458.013535] Key type ._llcrypt registered [ 5458.015844] Key type .llcrypt registered [ 5458.103514] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 5459.090660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5459.116149] alg: No test for adler32 (adler32-zlib) [ 5460.174393] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5460.362539] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 5461.992743] Key type lgssc registered [ 5463.593524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5472.565874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5472.606656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.056720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5478.921725] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5484.528586] LustreError: 176097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5484.537510] LustreError: 176097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5488.659088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.886577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5495.196707] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5502.782464] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5508.839414] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5515.429644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5515.680339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.785982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5522.059693] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5530.323263] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5542.889779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5542.904344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.934572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.434516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.455539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.456587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.498443] Lustre: Skipped 1 previous similar message [ 5549.549016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5549.556440] Lustre: Skipped 1 previous similar message [ 5554.679159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.698329] Lustre: Skipped 1 previous similar message [ 5555.169703] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5555.437722] Lustre: server umount lustre-OST0000 complete [ 5559.776204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.778378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5559.786802] Lustre: Skipped 1 previous similar message [ 5566.144119] Lustre: server umount lustre-MDT0000 complete [ 5570.019863] LustreError: 176092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.035607] LustreError: 176092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5570.047967] LustreError: 176078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757056 with bad export cookie 12600909597780769825 [ 5570.061061] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5570.061898] LustreError: 176078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5570.453663] Lustre: server umount lustre-MDT0001 complete [ 5582.433360] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5585.841401] Key type lgssc unregistered [ 5586.132257] LNet: 179636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5586.138841] LNetError: 179636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5586.151901] LNet: Removed LNI 192.168.201.154@tcp [ 5587.022439] Key type .llcrypt unregistered [ 5587.026580] Key type ._llcrypt unregistered [ 5611.863325] Key type ._llcrypt registered [ 5611.865017] Key type .llcrypt registered [ 5611.955391] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 5613.056968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5613.249937] alg: No test for adler32 (adler32-zlib) [ 5614.337405] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5614.793966] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 5616.535473] Key type lgssc registered [ 5617.673297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.331359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5629.340833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.725450] Lustre: *** cfs_fail_loc=135, val=0*** [ 5630.823649] LustreError: 181163:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5630.829950] LustreError: 181163:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.154@tcp: cfg command failed: rc = -2 [ 5630.837875] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5630.837875] [ 5630.847327] LustreError: MGC192.168.201.154@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5630.861994] LustreError: 181123:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5630.877070] LustreError: 181123:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5630.884872] LustreError: 181123:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5630.890027] LustreError: 181123:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5630.954908] Lustre: server umount lustre-MDT0000 complete [ 5630.956252] LustreError: 181123:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5638.822986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.217378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5643.977707] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5649.389651] LustreError: 181534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5649.414095] LustreError: 181534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5653.391272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.852833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5658.618200] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5664.991442] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5671.911474] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5674.465049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.472628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5674.482319] Lustre: Skipped 1 previous similar message [ 5674.507821] Lustre: Skipped 1 previous similar message [ 5679.595777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5679.607982] Lustre: Skipped 1 previous similar message [ 5680.618137] LustreError: 181529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.634346] LustreError: 181529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5680.899819] Lustre: server umount lustre-MDT0000 complete [ 5685.587651] LustreError: 181514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757171 with bad export cookie 8432655182729936796 [ 5685.613477] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.322684] Lustre: server umount lustre-MDT0001 complete [ 5697.654770] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5700.977253] Key type lgssc unregistered [ 5701.202668] LNet: 183860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5701.208195] LNetError: 183860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5701.216769] LNet: Removed LNI 192.168.201.154@tcp [ 5702.105637] Key type .llcrypt unregistered [ 5702.107921] Key type ._llcrypt unregistered [ 5724.081917] Key type ._llcrypt registered [ 5724.090420] Key type .llcrypt registered [ 5724.252944] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5724.827649] Key type .llcrypt unregistered [ 5724.833481] Key type ._llcrypt unregistered [ 5744.320752] Key type ._llcrypt registered [ 5744.322340] Key type .llcrypt registered [ 5744.390048] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 5745.219423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5745.352200] alg: No test for adler32 (adler32-zlib) [ 5746.381956] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5746.806527] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 5748.487209] Key type lgssc registered [ 5749.690538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5757.651618] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5757.667677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5773.279526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5778.400841] LustreError: 186011:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.154@tcp: failed processing log, type 4: rc = -110 [ 5805.151172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5812.841764] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5820.885940] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5827.121579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.670902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.725480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5831.517636] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5832.681509] LustreError: 186037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5832.700243] LustreError: 186037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5837.792942] LustreError: 186036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.865526] LustreError: 186037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.885823] LustreError: 186037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5840.621727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.158545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5846.876405] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5856.827777] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5861.063962] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5864.366422] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5878.754497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5878.768906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.795810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5882.339056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.344740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5887.461132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5887.472890] Lustre: Skipped 2 previous similar messages [ 5892.589355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5892.600526] Lustre: Skipped 1 previous similar message [ 5893.087161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5893.202928] Lustre: server umount lustre-OST0000 complete [ 5897.699270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.726206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.727441] Lustre: Skipped 1 previous similar message [ 5897.748179] Lustre: Skipped 1 previous similar message [ 5903.448949] Lustre: server umount lustre-MDT0000 complete [ 5907.764529] LustreError: 186017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757394 with bad export cookie 7795104224092597578 [ 5907.777327] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.782838] LustreError: 186017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5908.201154] Lustre: server umount lustre-MDT0001 complete [ 5921.403589] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 5924.183476] Key type lgssc unregistered [ 5924.435630] LNet: 190028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5924.440976] LNetError: 190028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5925.478218] LNet: Removed LNI 192.168.201.154@tcp [ 5926.187524] Key type .llcrypt unregistered [ 5926.190393] Key type ._llcrypt unregistered [ 5950.437187] Key type ._llcrypt registered [ 5950.439179] Key type .llcrypt registered [ 5950.529352] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 5952.067231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5952.109252] alg: No test for adler32 (adler32-zlib) [ 5953.228931] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5953.601200] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 5955.255472] Key type lgssc registered [ 5956.408277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5965.490188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5965.512729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.492196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.704274] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 5977.063993] LustreError: 191440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.079549] LustreError: 191440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5982.183417] LustreError: 191441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5986.100506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.763674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5992.671582] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6002.338539] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6008.016383] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6016.030173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.766361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.636542] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6032.715945] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6037.305848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6043.399080] Lustre: Failing over lustre-MDT0000 [ 6043.902933] Lustre: server umount lustre-MDT0000 complete [ 6044.853627] LustreError: 193936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.720540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.731530] LustreError: 191435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.763599] Lustre: Skipped 1 previous similar message [ 6047.810887] LustreError: 191435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6052.835381] LustreError: 191437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.858361] LustreError: 191437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6061.219517] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.233944] LustreError: 191436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6063.559265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.720237] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6064.085638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.148528] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6065.258540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6069.036739] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6069.255274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6069.271709] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6069.312474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6073.186880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6083.112804] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6097.899081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6097.917091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.929876] Lustre: Skipped 1 previous similar message [ 6097.933780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6098.927319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6098.934666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.947364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.873607] Lustre: server umount lustre-OST0000 complete [ 6110.178279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.187801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6110.222600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6110.222607] Lustre: Skipped 2 previous similar messages [ 6113.559279] Lustre: server umount lustre-MDT0000 complete [ 6115.300486] LustreError: 192168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.312915] LustreError: 192168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6117.162813] LustreError: 195025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757603 with bad export cookie 1078482234138700859 [ 6117.165317] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.167490] LustreError: 195025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6117.599559] Lustre: server umount lustre-MDT0001 complete [ 6128.821884] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 6131.434812] Key type lgssc unregistered [ 6131.703299] LNet: 196406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6131.721154] LNetError: 196406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6131.760062] LNet: Removed LNI 192.168.201.154@tcp [ 6132.873071] Key type .llcrypt unregistered [ 6132.879061] Key type ._llcrypt unregistered [ 6158.732645] Key type ._llcrypt registered [ 6158.742721] Key type .llcrypt registered [ 6158.951157] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 6160.838341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6160.930349] alg: No test for adler32 (adler32-zlib) [ 6162.218398] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 6162.612599] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 6164.423120] Key type lgssc registered [ 6166.603272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6175.984641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6176.004830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.720538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.923188] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6188.023775] LustreError: 197821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.036276] LustreError: 197821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6190.974388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.682798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6196.878599] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6206.356800] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6214.046563] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6222.997173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.334356] LustreError: 199641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.702513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.003078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6231.592666] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6240.647673] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6244.538719] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6251.093855] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6267.468693] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6277.091513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.108262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6277.115526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6277.125944] Lustre: Skipped 1 previous similar message [ 6282.214870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.222397] Lustre: Skipped 1 previous similar message [ 6283.151084] Lustre: server umount lustre-OST0000 complete [ 6287.328978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.337268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6287.343838] Lustre: Skipped 2 previous similar messages [ 6292.453141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.443410] Lustre: server umount lustre-MDT0000 complete [ 6297.568850] LustreError: 197820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.594386] LustreError: 197820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6299.303338] LustreError: 197801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757785 with bad export cookie 12075900597638915058 [ 6299.313921] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.326352] LustreError: 197801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6300.101441] Lustre: server umount lustre-MDT0001 complete [ 6313.092061] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 6316.594427] Key type lgssc unregistered [ 6316.966399] LNet: 201671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.973239] LNetError: 201671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6317.990148] LNet: Removed LNI 192.168.201.154@tcp [ 6318.955900] Key type .llcrypt unregistered [ 6318.959031] Key type ._llcrypt unregistered [ 6342.159263] Key type ._llcrypt registered [ 6342.163553] Key type .llcrypt registered [ 6342.386197] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 6344.674406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6344.736545] alg: No test for adler32 (adler32-zlib) [ 6346.252693] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 6346.601061] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 6348.303252] Key type lgssc registered [ 6350.708464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6358.911091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6358.961356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.647823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.659098] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6369.763395] LustreError: 202947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6369.794815] LustreError: 202947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6374.882102] LustreError: 202946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6375.482978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.827631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6381.009952] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6387.381482] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6392.804867] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6399.660927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.997298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.162233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6405.841141] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6415.395170] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6421.074523] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6434.755733] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6445.084033] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6448.427934] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6462.444759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6462.457698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6462.481894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.531939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.536929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.747946] Lustre: server umount lustre-OST0000 complete [ 6473.187394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6473.199823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6473.218490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6473.233425] Lustre: Skipped 1 previous similar message [ 6476.770231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.771290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6476.806067] Lustre: Skipped 1 previous similar message [ 6478.767626] Lustre: server umount lustre-MDT0000 complete [ 6481.895419] LustreError: 202947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.925954] LustreError: 202947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6483.123217] LustreError: 202928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757969 with bad export cookie 14379627755611630086 [ 6483.131377] LustreError: 202928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6483.132691] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.380335] Lustre: server umount lustre-MDT0001 complete [ 6496.234558] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 6499.229509] Key type lgssc unregistered [ 6499.721889] LNet: 207970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6499.732812] LNetError: 207970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6499.774916] LNet: Removed LNI 192.168.201.154@tcp [ 6500.690687] Key type .llcrypt unregistered [ 6500.696983] Key type ._llcrypt unregistered [ 6528.193765] Key type ._llcrypt registered [ 6528.195837] Key type .llcrypt registered [ 6528.341924] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 6529.761320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6529.867523] alg: No test for adler32 (adler32-zlib) [ 6530.906720] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 6531.224702] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 6532.919939] Key type lgssc registered [ 6534.060570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6541.373604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6541.389693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.026095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.538247] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6553.576952] LustreError: 209386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.612946] LustreError: 209386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6556.285594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.630571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6561.881211] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6570.163035] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6576.428278] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6582.477542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6582.800314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6587.944171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6589.911543] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6597.449981] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6613.485730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6613.494708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6613.512298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6616.546411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6616.551268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6616.556106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.205546] Lustre: server umount lustre-OST0000 complete [ 6623.202789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6623.213234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.231417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6623.238933] Lustre: Skipped 1 previous similar message [ 6626.784547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6626.790326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.265634] Lustre: server umount lustre-MDT0000 complete [ 6631.912461] LustreError: 209387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6631.943257] LustreError: 209387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6632.647289] LustreError: 209367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758118 with bad export cookie 16257472704532942564 [ 6632.675729] LustreError: 209367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6632.677344] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6633.509099] Lustre: server umount lustre-MDT0001 complete [ 6650.739775] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 6655.526460] Key type lgssc unregistered [ 6656.230637] LNet: 213073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6656.263402] LNetError: 213073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6657.351039] LNet: Removed LNI 192.168.201.154@tcp [ 6658.427391] Key type .llcrypt unregistered [ 6658.428653] Key type ._llcrypt unregistered [ 6679.482401] Key type ._llcrypt registered [ 6679.484585] Key type .llcrypt registered [ 6679.582374] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 6680.688101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6680.851914] alg: No test for adler32 (adler32-zlib) [ 6681.941864] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 6682.193621] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 6683.847183] Key type lgssc registered [ 6685.497956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6694.292540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6694.314386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.568865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6699.931307] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6705.122818] LustreError: 214341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.156266] LustreError: 214341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6707.936274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6708.417695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6713.736638] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6720.574332] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6727.146358] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6733.596511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.107960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.100135] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6740.281778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6748.110341] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.039556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.047557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6764.049188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6764.067932] Lustre: Skipped 1 previous similar message [ 6768.135112] Lustre: server umount lustre-OST0000 complete [ 6774.242880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.262932] Lustre: Skipped 1 previous similar message [ 6774.268871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6779.361352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6779.371675] Lustre: Skipped 2 previous similar messages [ 6784.483927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6786.015273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6786.215556] Lustre: server umount lustre-MDT0000 complete [ 6789.609899] LustreError: 214336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.631187] LustreError: 214336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6790.009641] LustreError: 214321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758276 with bad export cookie 3767981915277736573 [ 6790.022239] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6790.025600] LustreError: 214321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6790.427266] Lustre: server umount lustre-MDT0001 complete [ 6799.989576] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 6802.456765] Key type lgssc unregistered [ 6802.689976] LNet: 218024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6802.694586] LNetError: 218024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6802.709126] LNet: Removed LNI 192.168.201.154@tcp [ 6803.440976] Key type .llcrypt unregistered [ 6803.442672] Key type ._llcrypt unregistered [ 6820.702679] Key type ._llcrypt registered [ 6820.708767] Key type .llcrypt registered [ 6820.815491] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 6821.789253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6821.888071] alg: No test for adler32 (adler32-zlib) [ 6823.116645] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 6823.352567] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 6825.039473] Key type lgssc registered [ 6826.468190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6835.447175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6835.470939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.867501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6841.205964] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6847.470034] LustreError: 219293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.501068] LustreError: 219293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6850.062582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.598627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6855.599512] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6863.041668] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6868.693548] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6876.277882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6876.644471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6881.779441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6882.608958] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6889.902136] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6898.412805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.494526] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6898.780215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6904.982876] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 6914.177918] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6915.538398] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6927.581655] Lustre: Permanently deactivating lustre-OST0001 [ 6927.589840] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6937.713258] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6948.781769] Lustre: Permanently reactivating lustre-OST0001 [ 6948.789133] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6948.791409] Lustre: Skipped 2 previous similar messages [ 6954.563429] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.574389] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6954.581676] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6954.597368] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6954.626115] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6963.702148] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.711699] Lustre: Skipped 1 previous similar message [ 6963.719065] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6963.725581] Lustre: Skipped 1 previous similar message [ 6967.637541] Lustre: server umount lustre-OST0001 complete [ 6968.832198] LustreError: 221115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.858132] LustreError: 221115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6971.880697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.903865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.931022] Lustre: Skipped 1 previous similar message [ 6971.944054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.925358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.925491] LustreError: 221114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.927260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6973.930572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.977457] LustreError: 221114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6977.307764] Lustre: server umount lustre-OST0000 complete [ 6983.655975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6983.667228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.681291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6983.691591] Lustre: Skipped 1 previous similar message [ 6986.813756] Lustre: server umount lustre-MDT0000 complete [ 6989.298547] LustreError: 219289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6989.317752] LustreError: 219289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6990.391288] LustreError: 220027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758476 with bad export cookie 12438437233424295799 [ 6990.397695] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.399540] LustreError: 220027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6990.759617] Lustre: server umount lustre-MDT0001 complete [ 7000.858449] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 7003.367707] Key type lgssc unregistered [ 7003.657970] LNet: 224414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7003.670853] LNetError: 224414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7003.700690] LNet: Removed LNI 192.168.201.154@tcp [ 7004.425914] Key type .llcrypt unregistered [ 7004.427567] Key type ._llcrypt unregistered [ 7043.693639] Key type ._llcrypt registered [ 7043.697851] Key type .llcrypt registered [ 7043.787868] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 7044.488724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7044.604047] alg: No test for adler32 (adler32-zlib) [ 7045.836483] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 7046.066977] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 7047.808139] Key type lgssc registered [ 7049.017369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7056.335975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7056.351420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.533305] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7057.551803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7057.727290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.383535] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 7069.208568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.313624] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7069.351807] Lustre: 226619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7069.657392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7072.778867] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 7080.016316] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7085.203337] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7091.353228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7091.580466] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7091.777901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.140345] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 7100.121152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7105.204369] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7108.718915] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7114.162342] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7119.468937] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7128.826335] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7137.397499] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7157.574028] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7157.577506] Lustre: Skipped 2 previous similar messages [ 7192.991111] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7192.995612] Lustre: Skipped 3 previous similar messages [ 7259.626576] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7259.632521] Lustre: Skipped 8 previous similar messages [ 7279.591504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7279.596944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.602689] LustreError: Skipped 1 previous similar message [ 7279.603374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7279.616560] Lustre: Skipped 1 previous similar message [ 7279.638801] Lustre: Skipped 1 previous similar message [ 7284.710479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7284.716263] Lustre: Skipped 1 previous similar message [ 7285.118129] Lustre: server umount lustre-OST0000 complete [ 7289.826909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.827915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7289.844040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7289.861867] Lustre: Skipped 1 previous similar message [ 7295.048563] Lustre: server umount lustre-MDT0000 complete [ 7298.954422] LustreError: 229349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758785 with bad export cookie 6236668712989405425 [ 7298.967785] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7298.978987] LustreError: 229349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7299.303048] Lustre: server umount lustre-MDT0001 complete [ 7309.748300] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 7312.810736] Key type lgssc unregistered [ 7313.041836] LNet: 231457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7313.050136] LNetError: 231457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7313.077509] LNet: Removed LNI 192.168.201.154@tcp [ 7314.198461] Key type .llcrypt unregistered [ 7314.200807] Key type ._llcrypt unregistered [ 7337.431749] Key type ._llcrypt registered [ 7337.434431] Key type .llcrypt registered [ 7337.529750] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 7338.406273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7338.591821] alg: No test for adler32 (adler32-zlib) [ 7339.683358] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 7339.931596] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 7341.608630] Key type lgssc registered [ 7342.841472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7350.633912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7350.654481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7352.120940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7356.463354] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 7362.537570] LustreError: 232867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.576902] LustreError: 232867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7364.351816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7364.624951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7369.163607] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 7376.978652] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7382.386819] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7388.670558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.261440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7394.802995] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_default_debug -1 all [ 7397.365428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7400.917623] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7419.891145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7419.905660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.931731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7420.890359] Lustre: server umount lustre-OST0000 complete [ 7426.018090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7426.026521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7426.043652] Lustre: Skipped 1 previous similar message [ 7426.046921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7426.049788] Lustre: Skipped 1 previous similar message [ 7431.650686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7431.665615] Lustre: Skipped 2 previous similar messages [ 7431.841823] Lustre: server umount lustre-MDT0000 complete [ 7434.507604] LustreError: 232848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758920 with bad export cookie 7044291143492760091 [ 7434.513643] LustreError: 232848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7434.514116] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7434.820656] Lustre: server umount lustre-MDT0001 complete [ 7445.230230] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 7448.063082] Key type lgssc unregistered [ 7448.359834] LNet: 236564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7448.365662] LNetError: 236564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7449.382221] LNet: Removed LNI 192.168.201.154@tcp [ 7450.197582] Key type .llcrypt unregistered [ 7450.199331] Key type ._llcrypt unregistered [ 7472.246888] Key type ._llcrypt registered [ 7472.251938] Key type .llcrypt registered [ 7472.342595] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 7473.800321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7473.850625] alg: No test for adler32 (adler32-zlib) [ 7474.920562] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 7475.283572] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 7476.985698] Key type lgssc registered [ 7478.192085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7750.029614] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 7752.746890] Key type lgssc unregistered [ 7753.054534] LNet: 238596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7753.068093] LNetError: 238596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7753.094378] LNet: Removed LNI 192.168.201.154@tcp [ 7754.426695] Key type .llcrypt unregistered [ 7754.432559] Key type ._llcrypt unregistered [ 7780.337542] Key type ._llcrypt registered [ 7780.341161] Key type .llcrypt registered [ 7780.471646] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing set_hostid [ 7797.561809] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing load_modules_local [ 7798.827291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7798.911128] alg: No test for adler32 (adler32-zlib) [ 7799.961533] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 7800.380379] LNet: Added LNI 192.168.201.154@tcp [8/256/0/180] [ 7802.103572] Key type lgssc registered [ 7803.582451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7835.507883] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7508 sec ========== 22:02:00 (1786759320) [ 7837.128026] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:02:02 (1786759322) === [ 7840.728650] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:02:05 (1786759325) === [ 7883.391793] Lustre: DEBUG MARKER: oleg154-server.virtnet: executing unload_modules_local [ 7886.392789] Key type lgssc unregistered [ 7886.716506] LNet: 243844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7886.721609] LNetError: 243844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7886.737408] LNet: Removed LNI 192.168.201.154@tcp [ 7887.647535] Key type .llcrypt unregistered [ 7887.649796] Key type ._llcrypt unregistered