[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471373572 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002285] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.005006] kvm-guest: setup PV IPIs [ 0.007803] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011147] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014091] *** VALIDATE selinux *** [ 0.022391] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026781] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028076] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031011] *** VALIDATE tmpfs *** [ 0.032485] *** VALIDATE proc *** [ 0.034170] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037118] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044135] debug: unmapping init [mem 0xffffffffbe059000-0xffffffffbe060fff] [ 0.047135] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056325] rcu: Hierarchical SRCU implementation. [ 0.058502] smp: Bringing up secondary CPUs ... [ 0.059646] x86: Booting SMP configuration: [ 0.060033] .... node #0, CPUs: #1 #2 #3 [ 0.063377] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144886] node 0 deferred pages initialised in 77ms [ 0.149010] devtmpfs: initialized [ 0.150181] x86/mm: Memory block size: 128MB [ 0.152880] gcov: version magic: 0x41383552 [ 0.154180] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156266] pinctrl core: initialized pinctrl subsystem [ 0.157229] [ 0.157987] ************************************************************* [ 0.158022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159016] ** ** [ 0.160018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161017] ** ** [ 0.162014] ** This means that this kernel is built to expose internal ** [ 0.163014] ** IOMMU data structures, which may compromise security on ** [ 0.164018] ** your system. ** [ 0.165019] ** ** [ 0.166014] ** If you see this message and you are not debugging the ** [ 0.167017] ** kernel, report this immediately to your vendor! ** [ 0.168016] ** ** [ 0.169014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170017] ************************************************************* [ 0.171735] NET: Registered protocol family 16 [ 0.172466] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175474] cpuidle: using governor menu [ 0.176000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178535] PCI: Using configuration type 1 for base access [ 0.180178] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192092] cryptd: max_cpu_qlen set to 1000 [ 0.194440] ACPI: Added _OSI(Module Device) [ 0.196015] ACPI: Added _OSI(Processor Device) [ 0.198026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204300] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211227] ACPI: Interpreter enabled [ 0.212068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214014] ACPI: Using IOAPIC for interrupt routing [ 0.216091] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219443] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240474] acpiphp: Slot [2] registered [ 0.242192] acpiphp: Slot [5] registered [ 0.244144] acpiphp: Slot [6] registered [ 0.245186] acpiphp: Slot [7] registered [ 0.247141] acpiphp: Slot [8] registered [ 0.248142] acpiphp: Slot [9] registered [ 0.250116] acpiphp: Slot [10] registered [ 0.251124] acpiphp: Slot [3] registered [ 0.252101] acpiphp: Slot [4] registered [ 0.254113] acpiphp: Slot [11] registered [ 0.255105] acpiphp: Slot [12] registered [ 0.257129] acpiphp: Slot [13] registered [ 0.258109] acpiphp: Slot [14] registered [ 0.260111] acpiphp: Slot [15] registered [ 0.261105] acpiphp: Slot [16] registered [ 0.263177] acpiphp: Slot [17] registered [ 0.264100] acpiphp: Slot [18] registered [ 0.265089] acpiphp: Slot [19] registered [ 0.267105] acpiphp: Slot [20] registered [ 0.268105] acpiphp: Slot [21] registered [ 0.269100] acpiphp: Slot [22] registered [ 0.271120] acpiphp: Slot [23] registered [ 0.272112] acpiphp: Slot [24] registered [ 0.274091] acpiphp: Slot [25] registered [ 0.275092] acpiphp: Slot [26] registered [ 0.276094] acpiphp: Slot [27] registered [ 0.278113] acpiphp: Slot [28] registered [ 0.279119] acpiphp: Slot [29] registered [ 0.281110] acpiphp: Slot [30] registered [ 0.282081] acpiphp: Slot [31] registered [ 0.283097] PCI host bridge to bus 0000:00 [ 0.284018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.290018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.294091] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.296138] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298935] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301336] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.311016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315848] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.321017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.323018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329091] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334837] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.356016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366674] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.375019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418986] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.481812] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.496017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.505016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.536000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.547020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.555017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.587749] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.595017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.603022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.635114] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.641017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.649019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.686336] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.688507] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.691681] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694512] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697299] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.703164] iommu: Default domain type: Passthrough [ 0.704420] SCSI subsystem initialized [ 0.706172] ACPI: bus type USB registered [ 0.707127] usbcore: registered new interface driver usbfs [ 0.709087] usbcore: registered new interface driver hub [ 0.711141] usbcore: registered new device driver usb [ 0.714170] pps_core: LinuxPPS API ver. 1 registered [ 0.715007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.719079] PTP clock support registered [ 0.721071] EDAC MC: Ver: 3.0.0 [ 0.723125] PCI: Using ACPI for IRQ routing [ 0.725797] NetLabel: Initializing [ 0.726014] NetLabel: domain hash size = 128 [ 0.726983] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.728067] NetLabel: unlabeled traffic allowed by default [ 0.730523] vgaarb: loaded [ 0.732321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.734015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.742000] clocksource: Switched to clocksource kvm-clock [ 0.845234] VFS: Disk quotas dquot_6.6.0 [ 0.846954] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.849778] *** VALIDATE ramfs *** [ 0.851136] *** VALIDATE hugetlbfs *** [ 0.852777] pnp: PnP ACPI init [ 0.855332] pnp: PnP ACPI: found 6 devices [ 0.880888] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.884797] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.887261] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.888822] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.891252] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.893523] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.896690] NET: Registered protocol family 2 [ 0.899205] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.904329] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.908034] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.912949] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.916480] TCP: Hash tables configured (established 65536 bind 65536) [ 0.919740] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.923431] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926841] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.930466] NET: Registered protocol family 1 [ 0.933461] RPC: Registered named UNIX socket transport module. [ 0.935859] RPC: Registered udp transport module. [ 0.938191] RPC: Registered tcp transport module. [ 0.940198] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.942965] NET: Registered protocol family 44 [ 0.944417] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.946252] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.947867] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.949757] PCI: CLS 0 bytes, default 64 [ 0.951516] Unpacking initramfs... [ 2.305811] debug: unmapping init [mem 0xffff8ed63cc54000-0xffff8ed63ffbffff] [ 2.309578] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.311732] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.314663] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.793197] Initialise system trusted keyrings [ 2.794948] Key type blacklist registered [ 2.796850] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.805689] zbud: loaded [ 2.809034] *** VALIDATE nfs *** [ 2.810146] *** VALIDATE nfs4 *** [ 2.811676] pstore: using deflate compression [ 2.815509] Platform Keyring initialized [ 2.908621] NET: Registered protocol family 38 [ 2.910570] Key type asymmetric registered [ 2.912093] Asymmetric key parser 'x509' registered [ 2.913819] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.916790] io scheduler mq-deadline registered [ 2.918462] io scheduler kyber registered [ 2.920666] io scheduler bfq registered [ 2.922507] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.925347] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.927921] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.930605] ACPI: Power Button [PWRF] [ 2.935754] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.942442] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.952594] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.959186] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.974818] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.003011] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.031611] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.037413] Non-volatile memory driver v1.3 [ 3.038810] Linux agpgart interface v0.103 [ 3.073390] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.076000] vda: detected capacity change from 0 to 76779520 [ 3.090386] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.092967] vdb: detected capacity change from 0 to 1073741824 [ 3.106786] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.109256] vdc: detected capacity change from 0 to 2621440000 [ 3.123425] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.125899] vdd: detected capacity change from 0 to 2621440000 [ 3.142545] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.145725] vde: detected capacity change from 0 to 4294967296 [ 3.168152] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.171291] vdf: detected capacity change from 0 to 4294967296 [ 3.179133] libphy: Fixed MDIO Bus: probed [ 3.189579] usbcore: registered new interface driver usbserial_generic [ 3.192959] usbserial: USB Serial support registered for generic [ 3.195455] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.199703] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.201660] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.203967] mousedev: PS/2 mouse device common for all mice [ 3.207208] rtc_cmos 00:05: RTC can wake from S4 [ 3.210707] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.211123] rtc_cmos 00:05: registered as rtc0 [ 3.216745] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.218847] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.220175] intel_pstate: CPU model not supported [ 3.225553] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.228219] hid: raw HID events driver (C) Jiri Kosina [ 3.231436] usbcore: registered new interface driver usbhid [ 3.232851] usbhid: USB HID core driver [ 3.234113] drop_monitor: Initializing network drop monitor service [ 3.235888] Initializing XFRM netlink socket [ 3.237342] NET: Registered protocol family 10 [ 3.239809] Segment Routing with IPv6 [ 3.240965] NET: Registered protocol family 17 [ 3.242580] mpls_gso: MPLS GSO support [ 3.248501] RAS: Correctable Errors collector initialized. [ 3.250653] AVX version of gcm_enc/dec engaged. [ 3.252253] AES CTR mode by8 optimization enabled [ 3.317557] sched_clock: Marking stable (3317526209, 0)->(4339866501, -1022340292) [ 3.320747] registered taskstats version 1 [ 3.322575] Loading compiled-in X.509 certificates [ 3.324625] zswap: loaded using pool lzo/zbud [ 3.343098] Key type big_key registered [ 3.356489] Key type encrypted registered [ 3.358182] ima: No TPM chip found, activating TPM-bypass! [ 3.360282] ima: Allocated hash algorithm: sha1 [ 3.361982] ima: No architecture policies found [ 3.363701] evm: Initialising EVM extended attributes: [ 3.365478] evm: security.selinux [ 3.366770] evm: security.ima [ 3.367837] evm: security.capability [ 3.369185] evm: HMAC attrs: 0x1 [ 3.371378] rtc_cmos 00:05: setting system clock to 2026-09-07 17:26:20 UTC (1788801980) [ 3.377608] debug: unmapping init [mem 0xffffffffbf003000-0xffffffffbf1fffff] [ 3.380534] debug: unmapping init [mem 0xffffffffbdd82000-0xffffffffbe058fff] [ 3.390083] Write protecting the kernel read-only data: 28672k [ 3.393518] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.396651] debug: unmapping init [mem 0xffffffffbcd14000-0xffffffffbcdfffff] [ 3.429920] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.439358] systemd[1]: Detected virtualization kvm. [ 3.441360] systemd[1]: Detected architecture x86-64. [ 3.443642] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.471271] systemd[1]: No hostname configured. [ 3.473079] systemd[1]: Set hostname to . [ 3.475150] random: systemd: uninitialized urandom read (16 bytes read) [ 3.477561] systemd[1]: Initializing machine ID from random generator. [ 3.519545] random: ln: uninitialized urandom read (6 bytes read) [ 3.599616] random: systemd: uninitialized urandom read (16 bytes read) [ 3.602654] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.607137] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.613448] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.141993] device-mapper: uevent: version 1.0.3 [ 4.144554] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.808777] random: fast init done [ 4.833334] virtio_net virtio0 ens2: renamed from eth0 [ 4.893777] scsi host0: ata_piix [ 4.926311] scsi host1: ata_piix [ 4.927780] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.930252] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.145729] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.873354] random: crng init done [ 9.874676] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.185221] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.235744] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.474263] SELinux: Disabled at runtime. [ 11.534088] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.545483] systemd[1]: Detected virtualization kvm. [ 11.547520] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.019657] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.023198] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.028759] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.035349] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.038325] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.045896] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.055776] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK [ 12.134915] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.443566] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.765840] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.789439] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.887538] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.897065] EDAC sbridge: Ver: 1.1.2 [ 14.167901] Key type dns_resolver registered [ 14.470539] NFS: Registering the id_resolver key type [ 14.472552] Key type id_resolver registered [ 14.474104] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 39.032944] libcfs: loading out-of-tree module taints kernel. [ 39.056039] Key type ._llcrypt registered [ 39.057046] Key type .llcrypt registered [ 39.105853] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 52.264934] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 53.959572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.968913] alg: No test for adler32 (adler32-zlib) [ 55.354636] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 56.090819] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 57.807281] Key type lgssc registered [ 59.139186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.139027] hrtimer: interrupt took 11417483 ns [ 77.995939] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.364403] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 141.478704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.523516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.101851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.177863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.336339] Lustre: lustre-MDT0000: new disk, initializing [ 143.501236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.516420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.220730] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.115420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.281926] Lustre: 6486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.353463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.361947] Lustre: Skipped 1 previous similar message [ 163.473676] Lustre: lustre-MDT0001: new disk, initializing [ 163.608192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.642504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.654851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 168.453226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.840536] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.377679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.565291] Lustre: lustre-OST0000: new disk, initializing [ 184.568433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.572725] Lustre: 8424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.622868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.143419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.158448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.315546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.381926] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.264117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.401268] Lustre: lustre-OST0001: new disk, initializing [ 208.404155] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.409300] Lustre: 9495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.476510] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.761876] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.234517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 218.244248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 218.307318] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.955969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.387507] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.121190] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.214190] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 252.231449] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 254.670368] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 257.742429] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 259.757163] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 13:30:35 EDT 2026 [ 279.517789] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 281.215632] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 300.514721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.525387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.534161] Lustre: Skipped 2 previous similar messages [ 300.553780] Lustre: Skipped 3 previous similar messages [ 303.446824] Lustre: server umount lustre-MDT0000 complete [ 307.362786] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802284 with bad export cookie 10595751100951089511 [ 307.366104] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.375524] LustreError: 6478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 310.753534] LustreError: 6492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.758426] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.766069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.773207] LustreError: 6492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 310.807243] Lustre: Skipped 2 previous similar messages [ 312.805348] LustreError: 6492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.806718] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.833224] LustreError: 6492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 312.850059] Lustre: Skipped 2 previous similar messages [ 313.840757] Lustre: server umount lustre-MDT0001 complete [ 318.519415] Lustre: server umount lustre-OST0000 complete [ 323.312410] Lustre: server umount lustre-OST0001 complete [ 329.598913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 340.801852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 378.408095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.644604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 378.684050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 378.772586] Lustre: lustre-MDT0000: new disk, initializing [ 378.876842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.910131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.060911] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 394.785285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.935471] Lustre: 15051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 395.011490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 395.014949] Lustre: Skipped 1 previous similar message [ 395.124873] Lustre: lustre-MDT0001: new disk, initializing [ 395.227434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 395.257053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 395.263992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.290280] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 410.362094] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 417.137732] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.895789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.150190] Lustre: lustre-OST0000: new disk, initializing [ 425.153803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.159119] Lustre: 17021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.259934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.380891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.411720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.512439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.429363] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 443.589863] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.420823] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.648942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 457.297400] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 457.584440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.310325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.311510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.329841] LustreError: Skipped 1 previous similar message [ 462.332917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.383650] Lustre: Skipped 1 previous similar message [ 466.231418] Lustre: server umount lustre-OST0000 complete [ 472.036743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 472.045963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.060020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.065894] Lustre: Skipped 1 previous similar message [ 475.637483] LustreError: 18563:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8ed6bf1f6100) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 475.928867] Lustre: server umount lustre-MDT0000 complete [ 477.684601] LustreError: 15063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.714516] LustreError: 15063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 480.620599] LustreError: 17020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802457 with bad export cookie 10595751100951091044 [ 480.623609] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.630336] LustreError: 17020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 481.052928] Lustre: server umount lustre-MDT0001 complete [ 485.181601] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 13:34:21 (1788802461) [ 493.488388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.899676] LustreError: 19288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.968321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.172233] LustreError: 19289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.407589] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 509.053164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.417396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 509.420098] Lustre: Skipped 1 previous similar message [ 514.569956] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 522.820720] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 529.105176] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 537.760885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 546.820533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 558.972063] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.339049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.345319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.353941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.365104] Lustre: Skipped 2 previous similar messages [ 573.066768] Lustre: server umount lustre-OST0000 complete [ 580.578159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.715232] Lustre: server umount lustre-MDT0000 complete [ 585.697019] LustreError: 19289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.737573] LustreError: 19289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 587.476488] LustreError: 19269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802564 with bad export cookie 10595751100951091926 [ 587.481248] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.486334] LustreError: 19269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 587.807480] Lustre: server umount lustre-MDT0001 complete [ 600.161836] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 603.289826] Key type lgssc unregistered [ 603.789252] LNet: 22820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.796535] LNetError: 22820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 603.819570] LNet: Removed LNI 192.168.201.118@tcp [ 604.980172] Key type .llcrypt unregistered [ 604.984204] Key type ._llcrypt unregistered [ 634.102970] Key type ._llcrypt registered [ 634.106553] Key type .llcrypt registered [ 634.253119] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 635.994272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 636.048262] alg: No test for adler32 (adler32-zlib) [ 637.234211] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 637.569788] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 639.271211] Key type lgssc registered [ 640.537940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.071949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 650.096109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.590788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.063576] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 660.963988] LustreError: 24231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.995412] LustreError: 24231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 666.096700] LustreError: 24230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.799424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.510838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.908526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 684.128854] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 691.349352] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 699.044037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 699.597716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 706.868247] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 716.450553] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 721.173957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 734.689186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 734.695320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.710387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 736.440446] Lustre: server umount lustre-OST0000 complete [ 741.858975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.894796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 746.467046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 746.478468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.500698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 751.075159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 751.080370] Lustre: Skipped 2 previous similar messages [ 751.195573] Lustre: server umount lustre-MDT0000 complete [ 755.702727] LustreError: 24967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802732 with bad export cookie 11268052544453882066 [ 755.720531] LustreError: 24967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 755.725151] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.123234] Lustre: server umount lustre-MDT0001 complete [ 768.495556] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 771.920064] Key type lgssc unregistered [ 772.254059] LNet: 28212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.259240] LNetError: 28212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.275845] LNet: Removed LNI 192.168.201.118@tcp [ 773.294635] Key type .llcrypt unregistered [ 773.297340] Key type ._llcrypt unregistered [ 799.960550] Key type ._llcrypt registered [ 799.961985] Key type .llcrypt registered [ 800.099631] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 801.442460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.485691] alg: No test for adler32 (adler32-zlib) [ 802.575513] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 802.932966] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 804.703290] Key type lgssc registered [ 806.631451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 817.694623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 817.737306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.330024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.716539] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 828.899622] LustreError: 29626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.908392] LustreError: 29626:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 832.555551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.909573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 837.574161] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 846.362518] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 853.174354] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 870.582959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.913430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.060882] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 884.023349] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 889.346396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 897.508548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.524867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.548704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.554673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.556209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.559675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 899.589117] Lustre: Skipped 1 previous similar message [ 904.676078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 904.685734] Lustre: Skipped 1 previous similar message [ 909.795515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.327128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 911.548091] Lustre: server umount lustre-OST0000 complete [ 920.033094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 920.036397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.049204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.065717] Lustre: Skipped 1 previous similar message [ 920.077348] Lustre: Skipped 1 previous similar message [ 921.904736] Lustre: server umount lustre-MDT0000 complete [ 925.154888] LustreError: 29626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.176121] LustreError: 29626:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 926.326858] LustreError: 32776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802903 with bad export cookie 12519070556985936610 [ 926.330987] LustreError: 28937:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005a021e4f x1875695663877888/t0(0) o41->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 0 dl 1788802913 ref 1 fl Rpc:RESQU/200/0 rc -5/-107 job:'' uid:0 gid:0 projid:4294967295 [ 926.335924] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 926.339863] LustreError: 32776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 926.823717] Lustre: server umount lustre-MDT0001 complete [ 941.269503] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 944.176394] Key type lgssc unregistered [ 944.526336] LNet: 33527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.554156] LNetError: 33527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.570948] LNet: Removed LNI 192.168.201.118@tcp [ 945.556303] Key type .llcrypt unregistered [ 945.559879] Key type ._llcrypt unregistered [ 971.634341] Key type ._llcrypt registered [ 971.637868] Key type .llcrypt registered [ 971.761227] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 973.057673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.347659] alg: No test for adler32 (adler32-zlib) [ 974.390840] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 974.579557] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 976.232300] Key type lgssc registered [ 977.424241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 988.167872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 988.205337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.097461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.325771] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1000.436947] LustreError: 34937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.464462] LustreError: 34937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1005.549328] LustreError: 34938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.355556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.590085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1011.514080] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1020.530125] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1026.879704] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.884877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1035.179236] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1035.369230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.373354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1043.155905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1052.144474] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1064.401081] Lustre: server umount lustre-OST0000 complete [ 1072.097033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.100096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.116339] Lustre: Skipped 1 previous similar message [ 1077.217744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.224024] Lustre: Skipped 2 previous similar messages [ 1078.375957] Lustre: server umount lustre-MDT0000 complete [ 1082.345140] LustreError: 35666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.367878] LustreError: 35666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1082.754196] LustreError: 34919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803059 with bad export cookie 7251117945517921134 [ 1082.777169] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.285587] Lustre: server umount lustre-MDT0001 complete [ 1105.351751] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1108.645717] Key type lgssc unregistered [ 1108.908053] LNet: 38473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.921497] LNetError: 38473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.942626] LNet: Removed LNI 192.168.201.118@tcp [ 1109.892426] Key type .llcrypt unregistered [ 1109.896906] Key type ._llcrypt unregistered [ 1135.106807] Key type ._llcrypt registered [ 1135.109677] Key type .llcrypt registered [ 1135.235760] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1136.476926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.680940] alg: No test for adler32 (adler32-zlib) [ 1137.832914] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1138.302088] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1140.031385] Key type lgssc registered [ 1141.262503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1150.076933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1150.132726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.733870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.248368] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1161.188986] LustreError: 39889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.212789] LustreError: 39889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1165.398880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.655800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1170.745907] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1179.196772] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.956669] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1193.452668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1194.003746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.118264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1200.628540] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1208.745850] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1220.577119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1220.587051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.607627] Lustre: Skipped 1 previous similar message [ 1220.613734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1220.623214] Lustre: Skipped 1 previous similar message [ 1225.725501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1225.740292] Lustre: Skipped 2 previous similar messages [ 1230.832097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.835996] Lustre: Skipped 1 previous similar message [ 1233.375641] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1233.666658] Lustre: server umount lustre-OST0000 complete [ 1256.418599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.422188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.433945] Lustre: Skipped 1 previous similar message [ 1256.445473] Lustre: Skipped 3 previous similar messages [ 1259.507380] Lustre: server umount lustre-MDT0000 complete [ 1261.537511] LustreError: 39890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.556325] LustreError: 39890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1263.118193] LustreError: 39870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803240 with bad export cookie 16595902505585532729 [ 1263.124310] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.127058] LustreError: 39870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1269.755207] Lustre: server umount lustre-MDT0001 complete [ 1282.379202] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1285.586473] Key type lgssc unregistered [ 1285.964755] LNet: 43599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.972174] LNetError: 43599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.000755] LNet: Removed LNI 192.168.201.118@tcp [ 1287.131989] Key type .llcrypt unregistered [ 1287.139927] Key type ._llcrypt unregistered [ 1315.479881] Key type ._llcrypt registered [ 1315.482768] Key type .llcrypt registered [ 1315.589362] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1317.364608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1317.415675] alg: No test for adler32 (adler32-zlib) [ 1318.495554] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1318.782455] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1320.511400] Key type lgssc registered [ 1321.790443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1330.263905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1330.279520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.814749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.114888] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1340.912975] LustreError: 45010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.930330] LustreError: 45010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1346.023916] LustreError: 45011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.630308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.178328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.014632] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1362.608725] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1369.860658] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1377.704200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.180173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.249656] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1393.798794] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1401.727887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1411.115598] Lustre: server umount lustre-MDT0000 complete [ 1412.066451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.075351] LustreError: 45006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.101114] Lustre: Skipped 2 previous similar messages [ 1412.152744] LustreError: 45006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1415.425388] LustreError: 45005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.562142] LustreError: 44991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803392 with bad export cookie 18133098844992954587 [ 1415.565373] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.579426] LustreError: 44991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1417.193455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.217125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1420.193046] LustreError: 45005:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.194987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1420.216933] LustreError: 45005:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1422.043557] Lustre: server umount lustre-MDT0001 complete [ 1442.954609] Lustre: server umount lustre-OST0000 complete [ 1461.480529] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1464.379884] Key type lgssc unregistered [ 1464.697732] LNet: 48892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.712353] LNetError: 48892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.764755] LNet: Removed LNI 192.168.201.118@tcp [ 1466.720503] Key type .llcrypt unregistered [ 1466.723175] Key type ._llcrypt unregistered [ 1497.423330] Key type ._llcrypt registered [ 1497.426928] Key type .llcrypt registered [ 1497.564404] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1498.786941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1499.047394] alg: No test for adler32 (adler32-zlib) [ 1500.275838] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1500.518950] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1502.223238] Key type lgssc registered [ 1503.626765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1512.228772] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1512.251350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1527.903398] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1533.025092] LustreError: 50277:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 1559.775217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.622950] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1575.723356] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1743.329743] Lustre: server umount lustre-OST0000 complete [ 1762.101168] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1765.400776] Key type lgssc unregistered [ 1765.662054] LNet: 51952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.673087] LNetError: 51952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.700965] LNet: Removed LNI 192.168.201.118@tcp [ 1766.551757] Key type .llcrypt unregistered [ 1766.555368] Key type ._llcrypt unregistered [ 1792.987607] Key type ._llcrypt registered [ 1792.989188] Key type .llcrypt registered [ 1793.111814] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1794.230384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.438961] alg: No test for adler32 (adler32-zlib) [ 1795.525300] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1795.802534] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1797.479740] Key type lgssc registered [ 1798.694570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1807.190697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1807.212080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.835891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.111894] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1818.081060] LustreError: 53363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.097891] LustreError: 53363:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1823.200470] LustreError: 53364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.451651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.743439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1828.563622] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1837.198515] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1843.593455] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1850.965344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.489922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.559090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1858.255584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1867.210531] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1874.401636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.403967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1874.428461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1874.444775] Lustre: Skipped 1 previous similar message [ 1876.250525] Lustre: server umount lustre-OST0000 complete [ 1884.646625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.667959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.674691] Lustre: Skipped 2 previous similar messages [ 1884.691052] Lustre: Skipped 1 previous similar message [ 1886.718691] Lustre: server umount lustre-MDT0000 complete [ 1889.760812] LustreError: 53359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.780569] LustreError: 53359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1890.735109] LustreError: 53342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803867 with bad export cookie 9203917158931838866 [ 1890.746995] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.755906] LustreError: 53342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.284814] Lustre: server umount lustre-MDT0001 complete [ 1904.547633] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1907.946548] Key type lgssc unregistered [ 1908.286268] LNet: 56896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.299329] LNetError: 56896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.337104] LNet: Removed LNI 192.168.201.118@tcp [ 1909.597146] Key type .llcrypt unregistered [ 1909.598907] Key type ._llcrypt unregistered [ 1938.951649] Key type ._llcrypt registered [ 1938.955596] Key type .llcrypt registered [ 1939.075991] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1940.045990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.113924] alg: No test for adler32 (adler32-zlib) [ 1941.284238] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 1941.527541] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1943.191726] Key type lgssc registered [ 1944.758439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1953.523888] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1953.559701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1969.119431] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1974.240278] LustreError: 58282:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 2000.927312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.389566] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2015.540377] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2024.267763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.441175] Lustre: MGS: Not available for connect from 0@lo (not set up) [ 2025.185515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.307384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2030.530655] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2030.564485] LustreError: 58307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.573586] LustreError: 58307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2035.684736] LustreError: 58308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.707879] LustreError: 58308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2040.807151] LustreError: 59318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.853160] LustreError: 59318:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2041.839383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.309878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.755299] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2055.665360] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2062.519937] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2066.401863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2066.411346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.435641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.943513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.966165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.982548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.962684] Lustre: server umount lustre-OST0000 complete [ 2098.657174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.661132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.691585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.709180] Lustre: Skipped 2 previous similar messages [ 2100.963225] Lustre: server umount lustre-MDT0000 complete [ 2103.785804] LustreError: 60667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.819870] LustreError: 60667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2104.935474] LustreError: 58290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804082 with bad export cookie 14937464801222596945 [ 2104.937102] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.942083] LustreError: 58290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2111.321820] Lustre: server umount lustre-MDT0001 complete [ 2124.838936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2128.619669] Key type lgssc unregistered [ 2128.956820] LNet: 62039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.983071] LNetError: 62039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.005640] LNet: Removed LNI 192.168.201.118@tcp [ 2130.254155] Key type .llcrypt unregistered [ 2130.256653] Key type ._llcrypt unregistered [ 2158.850102] Key type ._llcrypt registered [ 2158.852026] Key type .llcrypt registered [ 2159.015179] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2160.752553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.859080] alg: No test for adler32 (adler32-zlib) [ 2161.980410] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2162.221064] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2163.978161] Key type lgssc registered [ 2165.428774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2175.204160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2175.217226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.621620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.858768] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2186.209341] LustreError: 63448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.237357] LustreError: 63448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2191.333460] LustreError: 63449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.418581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.876533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.364330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2207.487806] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2214.280838] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.623523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2223.063255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.814541] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2240.332229] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2246.654243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2260.487267] Lustre: server umount lustre-OST0000 complete [ 2264.565210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2264.573222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.613638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.111505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.117788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.236332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.243174] Lustre: Skipped 2 previous similar messages [ 2277.362767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.381242] Lustre: Skipped 1 previous similar message [ 2278.367513] Lustre: 62758:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804239/real 1788804239] req@0000000023fcac9f x1875697088888704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788804255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2278.371062] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2278.429843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.311451] Lustre: server umount lustre-MDT0000 complete [ 2279.401256] Lustre: 62761:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804240/real 1788804240] req@000000005c56da94 x1875697088888960/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788804256 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2283.488712] LustreError: 63445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.494664] Lustre: 62760:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804244/real 1788804244] req@000000004fc7cc61 x1875697088890624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788804260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2283.524583] LustreError: 63445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2285.064693] LustreError: 65951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804262 with bad export cookie 3873374031903062051 [ 2285.077266] LustreError: 65951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2285.077497] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.566439] Lustre: server umount lustre-MDT0001 complete [ 2300.263480] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2303.628801] Key type lgssc unregistered [ 2304.153720] LNet: 67007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.171902] LNetError: 67007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.193311] LNet: Removed LNI 192.168.201.118@tcp [ 2306.330862] Key type .llcrypt unregistered [ 2306.332622] Key type ._llcrypt unregistered [ 2337.908328] Key type ._llcrypt registered [ 2337.910772] Key type .llcrypt registered [ 2338.028248] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2339.483843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2339.518993] alg: No test for adler32 (adler32-zlib) [ 2340.674593] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2341.067293] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2342.848766] Key type lgssc registered [ 2344.107735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.531605] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:05:32 (1788804332) [ 2364.418471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2364.440393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.981212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.042121] LustreError: 68707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.060314] LustreError: 68707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2371.734842] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2376.175596] LustreError: 68707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.283617] LustreError: 68708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.631380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.143945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2387.504663] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2396.551765] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2403.183293] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2412.820501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.509874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2421.292581] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2432.147590] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.037418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2445.528509] Lustre: Failing over lustre-MDT0000 [ 2445.945815] Lustre: server umount lustre-MDT0000 complete [ 2446.308679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.317963] LustreError: 68702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.334890] Lustre: Skipped 2 previous similar messages [ 2446.345269] LustreError: 68702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2451.432538] LustreError: 68703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.470843] LustreError: 68703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2457.062442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.255458] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.446712] Lustre: *** cfs_fail_loc=135, val=0*** [ 2457.598540] LustreError: 71822:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2457.607947] LustreError: 71822:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -2 [ 2457.615250] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2457.615250] [ 2457.627996] LustreError: MGC192.168.201.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2457.651075] LustreError: 71811:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2457.659392] LustreError: 71811:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2457.672443] LustreError: 71811:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2457.685292] LustreError: 71811:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2457.710454] Lustre: server umount lustre-MDT0000 complete [ 2457.712448] LustreError: 71811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2459.880516] LustreError: 68702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.909418] LustreError: 68702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2465.832371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.986457] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.157251] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (not set up) [ 2466.353169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.432736] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2470.187673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2471.416868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2471.458682] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2471.525720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2472.183782] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2481.636943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.659233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.775152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.787671] Lustre: Skipped 2 previous similar messages [ 2487.311235] Lustre: server umount lustre-OST0000 complete [ 2496.992867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2496.997716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.036152] Lustre: Skipped 1 previous similar message [ 2497.044030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2497.055202] Lustre: Skipped 1 previous similar message [ 2498.386114] Lustre: server umount lustre-MDT0000 complete [ 2502.113959] LustreError: 68708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.147599] LustreError: 68708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2503.171852] LustreError: 71314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804480 with bad export cookie 1653437232830148857 [ 2503.183667] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.186901] LustreError: 71314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2503.630185] Lustre: server umount lustre-MDT0001 complete [ 2517.230267] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2520.307756] Key type lgssc unregistered [ 2520.559430] LNet: 73602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.568205] LNetError: 73602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.596297] LNet: Removed LNI 192.168.201.118@tcp [ 2521.787788] Key type .llcrypt unregistered [ 2521.794485] Key type ._llcrypt unregistered [ 2550.489604] Key type ._llcrypt registered [ 2550.491922] Key type .llcrypt registered [ 2550.597724] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2551.873378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.995856] alg: No test for adler32 (adler32-zlib) [ 2553.192224] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2553.411110] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2555.111212] Key type lgssc registered [ 2556.409880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2565.791489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2565.820244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.478297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2573.217301] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2577.889242] LustreError: 75011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.903717] LustreError: 75011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2583.015962] LustreError: 75012:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.078889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.638242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2589.228805] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2597.376280] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2603.489663] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2611.970981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.336241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.358458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2621.246710] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2630.856879] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2637.489518] Lustre: Failing over lustre-MDT0000 [ 2637.762973] Lustre: server umount lustre-MDT0000 complete [ 2638.579203] LustreError: 76652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.328877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.681137] LustreError: 76652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.736312] LustreError: 76652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2647.480401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.719481] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.898744] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2648.021742] LustreError: 78101:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2648.027671] LustreError: 78101:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -9 [ 2648.033827] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2648.033827] [ 2648.043513] LustreError: MGC192.168.201.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2648.053785] LustreError: 78089:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2648.060091] LustreError: 78089:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2648.065671] LustreError: 78089:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2648.073868] LustreError: 78089:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2648.091672] Lustre: server umount lustre-MDT0000 complete [ 2648.094663] LustreError: 78089:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2648.800849] LustreError: 75006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.818305] LustreError: 75006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2656.612938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.841985] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2656.860389] LustreError: 75003:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2657.142993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.221057] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2660.063488] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2662.385411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2662.428571] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2662.492617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2662.676881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2672.639765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.658453] Lustre: Skipped 2 previous similar messages [ 2672.663676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.760505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2677.770528] Lustre: Skipped 1 previous similar message [ 2678.651478] Lustre: server umount lustre-OST0000 complete [ 2682.849656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2682.851265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.867357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.867366] Lustre: Skipped 1 previous similar message [ 2682.906160] Lustre: Skipped 2 previous similar messages [ 2687.972906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.987635] Lustre: Skipped 2 previous similar messages [ 2688.671295] Lustre: server umount lustre-MDT0000 complete [ 2692.738220] LustreError: 74992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804669 with bad export cookie 14787336909799282216 [ 2692.744111] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2692.748704] LustreError: 74992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2693.240319] Lustre: server umount lustre-MDT0001 complete [ 2711.076887] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2714.553457] Key type lgssc unregistered [ 2714.819898] LNet: 79901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2714.822824] LNetError: 79901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2714.844702] LNet: Removed LNI 192.168.201.118@tcp [ 2715.759966] Key type .llcrypt unregistered [ 2715.762597] Key type ._llcrypt unregistered [ 2732.397543] Key type ._llcrypt registered [ 2732.399219] Key type .llcrypt registered [ 2732.598061] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2732.829654] Key type .llcrypt unregistered [ 2732.832721] Key type ._llcrypt unregistered [ 2767.180812] Key type ._llcrypt registered [ 2767.186365] Key type .llcrypt registered [ 2767.357930] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2769.116912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2769.152695] alg: No test for adler32 (adler32-zlib) [ 2770.392255] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2770.779802] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2772.583270] Key type lgssc registered [ 2774.140725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2784.825994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2784.841025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.616192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.686301] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2796.001681] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.024281] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2799.082707] LustreError: 81802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.199270] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.111074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.816829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.511946] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2824.764452] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2832.700213] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2841.374968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2841.747646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.920232] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2857.224867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2858.972554] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2870.757898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2870.767096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.780696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.800773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2872.821781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.830831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2872.854544] Lustre: Skipped 1 previous similar message [ 2877.153760] Lustre: server umount lustre-OST0000 complete [ 2882.531476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2882.547085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.576294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2888.161870] LustreError: 81797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.194658] LustreError: 81797:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2888.311605] Lustre: server umount lustre-MDT0000 complete [ 2892.579286] LustreError: 83649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804869 with bad export cookie 1578864510647746392 [ 2892.593343] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.598922] LustreError: 83649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2892.930124] Lustre: server umount lustre-MDT0001 complete [ 2912.205817] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2918.655696] Key type lgssc unregistered [ 2919.584133] LNet: 85340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.609239] LNetError: 85340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.655910] LNet: Removed LNI 192.168.201.118@tcp [ 2921.524598] Key type .llcrypt unregistered [ 2921.540016] Key type ._llcrypt unregistered [ 2951.400368] Key type ._llcrypt registered [ 2951.403433] Key type .llcrypt registered [ 2951.553533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2952.764863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2952.806862] alg: No test for adler32 (adler32-zlib) [ 2954.133305] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 2954.450240] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2956.135495] Key type lgssc registered [ 2957.640621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2966.655192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2966.681296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.199311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.261964] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2978.786785] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.803299] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2983.357201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.800114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2989.383965] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2998.801674] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3005.534367] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3014.246449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3014.612697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3021.368292] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3030.121700] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3038.188214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3048.416886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3048.420583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.429459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3048.466814] Lustre: Skipped 1 previous similar message [ 3053.292734] Lustre: server umount lustre-OST0000 complete [ 3058.657749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.675580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3058.680881] Lustre: Skipped 2 previous similar messages [ 3063.779783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3063.788855] Lustre: Skipped 1 previous similar message [ 3063.911749] Lustre: server umount lustre-MDT0000 complete [ 3067.874294] LustreError: 86744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.903433] LustreError: 86744:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3068.040177] LustreError: 86729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805045 with bad export cookie 17655281546967787835 [ 3068.061762] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.648291] Lustre: server umount lustre-MDT0001 complete [ 3080.671097] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3084.211162] Key type lgssc unregistered [ 3084.470609] LNet: 90307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3084.480155] LNetError: 90307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3084.494578] LNet: Removed LNI 192.168.201.118@tcp [ 3085.630858] Key type .llcrypt unregistered [ 3085.637669] Key type ._llcrypt unregistered [ 3113.829799] Key type ._llcrypt registered [ 3113.831948] Key type .llcrypt registered [ 3113.988357] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3115.478188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3115.615658] alg: No test for adler32 (adler32-zlib) [ 3116.730943] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 3116.967621] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3118.722262] Key type lgssc registered [ 3120.119949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3129.436799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3129.485028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.070288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.263973] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3141.611602] LustreError: 91714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.635181] LustreError: 91714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3146.469281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.732073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3146.929619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3152.313849] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3161.505265] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3168.172411] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3176.185120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3176.513849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.862492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3192.957940] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3201.087629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3221.473923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.477668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3221.499035] Lustre: Skipped 1 previous similar message [ 3221.519737] Lustre: Skipped 1 previous similar message [ 3224.422715] Lustre: server umount lustre-OST0000 complete [ 3229.152212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3229.162680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.171260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.712723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.721396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.736430] Lustre: Skipped 1 previous similar message [ 3234.979852] Lustre: server umount lustre-MDT0000 complete [ 3236.833300] LustreError: 91715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.854697] LustreError: 91715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3238.830780] LustreError: 91697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805215 with bad export cookie 13873660695748259584 [ 3238.843576] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.847438] LustreError: 91697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3239.286952] Lustre: server umount lustre-MDT0001 complete [ 3251.807505] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3254.948996] Key type lgssc unregistered [ 3255.281963] LNet: 95270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3255.287983] LNetError: 95270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3256.357910] LNet: Removed LNI 192.168.201.118@tcp [ 3257.311968] Key type .llcrypt unregistered [ 3257.314135] Key type ._llcrypt unregistered [ 3282.217171] Key type ._llcrypt registered [ 3282.219165] Key type .llcrypt registered [ 3282.339402] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3283.623978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3283.698238] alg: No test for adler32 (adler32-zlib) [ 3285.099756] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 3285.386769] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3287.071239] Key type lgssc registered [ 3288.145447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3296.561452] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3296.582029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.159466] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3317.279347] LustreError: 96651:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 3343.967594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.395665] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3359.606386] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3365.113151] Lustre: server umount lustre-OST0000 complete [ 3374.966134] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:22:31 (1788805351) [ 3382.653286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3383.060110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.392648] LustreError: 98291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.426466] LustreError: 98291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3388.712455] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3393.506232] LustreError: 98292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.569419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.996891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3403.343880] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3412.064749] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3419.045226] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3426.370615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3426.713286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.800274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3434.652634] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3441.971363] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3454.436904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.444000] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.468804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3458.533814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.534868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.536346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.626563] Lustre: server umount lustre-OST0000 complete [ 3464.673248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3464.684241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.701826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.712609] Lustre: Skipped 1 previous similar message [ 3468.771173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.771668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.793541] Lustre: Skipped 1 previous similar message [ 3470.694238] Lustre: server umount lustre-MDT0000 complete [ 3473.889735] LustreError: 99265:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.903192] LustreError: 99265:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3474.533415] LustreError: 100955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805451 with bad export cookie 5520167000812537269 [ 3474.555084] LustreError: 100955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3474.555415] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.050392] Lustre: server umount lustre-MDT0001 complete [ 3485.971404] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3488.905319] Key type lgssc unregistered [ 3489.205731] LNet: 101993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.214324] LNetError: 101993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.231309] LNet: Removed LNI 192.168.201.118@tcp [ 3490.063319] Key type .llcrypt unregistered [ 3490.065497] Key type ._llcrypt unregistered [ 3530.736987] Key type ._llcrypt registered [ 3530.738670] Key type .llcrypt registered [ 3530.854300] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3531.601870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3531.636101] alg: No test for adler32 (adler32-zlib) [ 3532.702769] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 3532.908504] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3534.591601] Key type lgssc registered [ 3535.974936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3544.433376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3544.464540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.963436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.749337] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3556.326943] LustreError: 104168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.333908] LustreError: 104168:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3558.527726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.836072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3563.117828] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3569.786104] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3575.582312] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3582.618241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.057500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.133677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3591.281420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3597.362020] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3607.009271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3607.019174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.026966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.037292] Lustre: Skipped 1 previous similar message [ 3612.136499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.161486] Lustre: Skipped 2 previous similar messages [ 3612.603456] Lustre: server umount lustre-OST0000 complete [ 3617.251498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.266559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.274295] Lustre: Skipped 1 previous similar message [ 3622.369723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.381321] Lustre: Skipped 1 previous similar message [ 3622.812876] Lustre: server umount lustre-MDT0000 complete [ 3626.092642] LustreError: 104150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805603 with bad export cookie 2089664874585753078 [ 3626.099262] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3626.103063] LustreError: 104150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3626.515312] Lustre: server umount lustre-MDT0001 complete [ 3638.332072] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3641.209454] Key type lgssc unregistered [ 3641.585516] LNet: 107700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3641.590420] LNetError: 107700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3641.613291] LNet: Removed LNI 192.168.201.118@tcp [ 3642.454129] Key type .llcrypt unregistered [ 3642.455368] Key type ._llcrypt unregistered [ 3661.487835] Key type ._llcrypt registered [ 3661.489836] Key type .llcrypt registered [ 3661.636217] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3662.926084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.992706] alg: No test for adler32 (adler32-zlib) [ 3664.088693] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 3664.367878] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3666.127208] Key type lgssc registered [ 3667.348304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3675.778853] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3675.798389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.424323] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3696.543759] LustreError: 108939:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 3723.167142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.109928] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3734.182898] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3740.088930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.364103] LustreError: MGC192.168.201.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3740.372894] LustreError: 109964:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3740.381241] LustreError: 109964:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3740.391230] LustreError: 109964:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3740.454767] Lustre: server umount lustre-MDT0000 complete [ 3740.456692] LustreError: 109964:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3756.124405] Lustre: server umount lustre-OST0000 complete [ 3765.866176] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3776.243239] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3808.790085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.029635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3809.060930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3809.148946] Lustre: lustre-MDT0000: new disk, initializing [ 3809.192072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.212189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3813.319584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3823.719767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3823.845655] Lustre: 112754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3823.969765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3823.976444] Lustre: Skipped 1 previous similar message [ 3824.102483] Lustre: lustre-MDT0001: new disk, initializing [ 3824.229120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.271781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3824.293877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3828.677403] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3837.812661] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3843.820966] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3850.012624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3850.305770] Lustre: lustre-OST0000: new disk, initializing [ 3850.310891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3850.315426] Lustre: 114721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3850.383516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.357856] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3857.996839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3858.009374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3858.208229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3863.665772] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3868.190328] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3868.387501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3872.225099] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3872.339213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3873.764237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.772154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.777111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3873.788041] Lustre: Skipped 1 previous similar message [ 3878.881743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3878.886992] Lustre: Skipped 2 previous similar messages [ 3884.004667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.095117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3888.356612] Lustre: server umount lustre-OST0000 complete [ 3894.243755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.248343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3894.259615] Lustre: Skipped 1 previous similar message [ 3894.262186] Lustre: Skipped 1 previous similar message [ 3897.603446] Lustre: server umount lustre-MDT0000 complete [ 3900.730781] LustreError: 112746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788805877 with bad export cookie 17798056222291030325 [ 3900.742257] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.744549] LustreError: 112746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3901.062764] Lustre: server umount lustre-MDT0001 complete [ 3908.878682] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:31:25 (1788805885) [ 3911.908310] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3928.409498] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3937.781899] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3972.825192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.034521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3973.054735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3973.134635] Lustre: lustre-MDT0000: new disk, initializing [ 3973.211648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.240103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3976.984462] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3985.488279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.580582] Lustre: 119621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3985.629261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3985.631265] Lustre: Skipped 1 previous similar message [ 3985.684571] Lustre: lustre-MDT0001: new disk, initializing [ 3985.763436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3985.793200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3985.807698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3989.603315] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3999.269727] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4004.632334] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4011.197165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.483346] Lustre: lustre-OST0000: new disk, initializing [ 4011.486705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4011.495518] Lustre: 121589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4011.576640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4017.494699] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4020.779238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.798629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4020.949378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4026.806072] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4031.490110] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4031.741565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4035.840732] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4036.049914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4041.697052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4041.721389] LustreError: Skipped 1 previous similar message [ 4041.728724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.750160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4041.758767] Lustre: Skipped 1 previous similar message [ 4044.219233] Lustre: server umount lustre-OST0000 complete [ 4051.937082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.940897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.946304] Lustre: Skipped 2 previous similar messages [ 4051.957457] Lustre: Skipped 1 previous similar message [ 4053.844983] Lustre: server umount lustre-MDT0000 complete [ 4057.063467] LustreError: 120688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.073573] LustreError: 120424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806034 with bad export cookie 17798056222291031207 [ 4057.084657] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4057.086402] LustreError: 120688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4057.134278] LustreError: 120424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4057.516106] Lustre: server umount lustre-MDT0001 complete [ 4064.007316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.434020] LustreError: 123805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.523122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.289496] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4069.859502] LustreError: 123806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.954619] LustreError: 123805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.551898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.914368] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4083.625400] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4088.169384] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4093.845038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4099.348370] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4105.650109] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4116.268252] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4120.038135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.038794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4120.045900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.048641] Lustre: Skipped 1 previous similar message [ 4120.063565] LustreError: Skipped 2 previous similar messages [ 4125.814075] Lustre: server umount lustre-OST0000 complete [ 4130.276298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.751980] Lustre: server umount lustre-MDT0000 complete [ 4139.118782] LustreError: 123786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806116 with bad export cookie 17798056222291032089 [ 4139.126515] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.385411] Lustre: server umount lustre-MDT0001 complete [ 4148.574935] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4150.540498] Key type lgssc unregistered [ 4150.795056] LNet: 127433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4150.807507] LNetError: 127433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4151.849115] LNet: Removed LNI 192.168.201.118@tcp [ 4152.613818] Key type .llcrypt unregistered [ 4152.616228] Key type ._llcrypt unregistered [ 4168.433965] Key type ._llcrypt registered [ 4168.441315] Key type .llcrypt registered [ 4168.567271] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4182.516704] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4183.346798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4183.451077] alg: No test for adler32 (adler32-zlib) [ 4184.398978] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 4184.567813] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4186.183402] Key type lgssc registered [ 4187.040918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4213.803786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4213.831262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4214.996986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4215.020403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4215.076781] Lustre: lustre-MDT0000: new disk, initializing [ 4215.160506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.220064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4218.858709] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4227.646713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.748523] Lustre: 130583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4227.817241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4227.820923] Lustre: Skipped 1 previous similar message [ 4227.873464] Lustre: lustre-MDT0001: new disk, initializing [ 4227.989345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.039358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4228.051838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4231.761899] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4239.666203] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4243.728290] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4249.005595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4249.219657] Lustre: lustre-OST0000: new disk, initializing [ 4249.222296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4249.228630] Lustre: 132551:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4249.288877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.525468] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4258.342911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4258.354494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4258.404854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4261.924896] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4265.893900] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4266.061892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4269.484725] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4269.671267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4273.638957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4273.645383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4273.656039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4273.662946] Lustre: Skipped 1 previous similar message [ 4277.460469] Lustre: server umount lustre-OST0000 complete [ 4283.873898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.888348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.889702] Lustre: Skipped 2 previous similar messages [ 4283.900982] Lustre: Skipped 1 previous similar message [ 4286.238082] Lustre: server umount lustre-MDT0000 complete [ 4288.870303] LustreError: 132550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806265 with bad export cookie 3873694632208613455 [ 4288.875234] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4288.885240] LustreError: 132550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4289.126709] Lustre: server umount lustre-MDT0001 complete [ 4295.441697] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:37:52 (1788806272) [ 4300.955901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.450496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.408685] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4306.924209] LustreError: 134960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.943184] LustreError: 134960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4311.009280] LustreError: 134959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.657936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4313.002837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.576248] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4322.855799] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4327.667668] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4331.489988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.490913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4331.503630] Lustre: Skipped 1 previous similar message [ 4331.514192] Lustre: Skipped 1 previous similar message [ 4335.383364] Lustre: server umount lustre-MDT0000 complete [ 4336.609979] LustreError: 134955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.629297] LustreError: 134955:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4338.671530] LustreError: 134954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4338.956658] LustreError: 134941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806316 with bad export cookie 3873694632208614344 [ 4338.961359] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4338.974775] LustreError: 134941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4339.274282] Lustre: server umount lustre-MDT0001 complete [ 4346.197797] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:38:42 (1788806322) [ 4351.217151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4366.751824] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4371.871668] LustreError: 137373:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 4397.599503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.228148] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4407.872835] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4409.615479] Lustre: server umount lustre-OST0000 complete [ 4416.379356] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:39:53 (1788806393) [ 4421.377218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.669126] LustreError: 138819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.686374] LustreError: 138819:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4421.737861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4425.556300] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4431.329834] LustreError: 138819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.344681] LustreError: 138819:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4433.488802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.743671] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4444.292306] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4449.089314] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4454.361212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.580448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.583675] Lustre: Skipped 1 previous similar message [ 4458.918269] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4465.203253] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.415506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.419289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.433774] Lustre: Skipped 2 previous similar messages [ 4477.440686] Lustre: Skipped 1 previous similar message [ 4482.531237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.534993] Lustre: Skipped 3 previous similar messages [ 4482.639977] Lustre: server umount lustre-MDT0000 complete [ 4485.449982] LustreError: 138801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806462 with bad export cookie 3873694632208615009 [ 4485.453298] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4485.460452] LustreError: 138801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4485.751738] Lustre: server umount lustre-MDT0001 complete [ 4498.928514] Lustre: server umount lustre-OST0000 complete [ 4505.576354] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:41:22 (1788806482) [ 4510.666355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.034591] LustreError: 142472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.606893] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4520.207967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.382519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4520.397314] Lustre: Skipped 1 previous similar message [ 4523.235086] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4528.537917] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4532.736526] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4537.850533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4543.486420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4548.860693] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4552.617320] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4556.727027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4556.994628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4560.230701] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4560.403626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4565.984202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4565.990498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.013818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4567.013020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4576.223132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4576.414664] Lustre: server umount lustre-OST0000 complete [ 4581.856190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4581.861072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.869593] Lustre: Skipped 1 previous similar message [ 4582.368467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.371624] Lustre: Skipped 6 previous similar messages [ 4585.599228] Lustre: server umount lustre-MDT0000 complete [ 4587.492501] LustreError: 142473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.509638] LustreError: 142473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4588.614296] LustreError: 142454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806565 with bad export cookie 3873694632208616080 [ 4588.618534] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.623581] LustreError: 142454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4588.887380] Lustre: server umount lustre-MDT0001 complete [ 4595.760647] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:42:52 (1788806572) [ 4601.353478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4616.864111] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4621.984971] LustreError: 146658:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 4652.505932] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4658.937394] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4664.056314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.332345] LustreError: 146683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.390335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.396991] Lustre: Skipped 2 previous similar messages [ 4664.432974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4667.784577] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4674.530176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.178562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4683.362838] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4687.750265] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4690.779476] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4690.920943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4694.245656] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4694.372640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4697.572444] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.574221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.591091] Lustre: Skipped 2 previous similar messages [ 4697.601952] Lustre: Skipped 2 previous similar messages [ 4710.367131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4710.498521] Lustre: server umount lustre-MDT0000 complete [ 4713.289857] LustreError: 146666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806690 with bad export cookie 3873694632208617102 [ 4713.299908] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4713.300342] LustreError: 146666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4713.575501] Lustre: server umount lustre-MDT0001 complete [ 4726.523394] Lustre: server umount lustre-OST0000 complete [ 4732.325581] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:45:09 (1788806709) [ 4736.489939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4752.032667] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4757.152132] LustreError: 150851:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 4786.883892] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4791.264315] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4795.480225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.736381] LustreError: 150876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.751093] LustreError: 150876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4795.840761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4798.364157] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4804.356412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4807.557210] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4812.839727] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4817.047492] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4821.882701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4821.999327] Lustre: lustre-OST0001: new disk, initializing [ 4822.000975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4822.003431] Lustre: 153721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4825.814595] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4828.186973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4828.195172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4828.260062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4832.301205] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4835.226769] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4835.385419] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4838.464847] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4838.609796] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4840.422331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4840.431707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.452922] Lustre: Skipped 1 previous similar message [ 4840.459903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4840.466779] Lustre: Skipped 6 previous similar messages [ 4854.239174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4854.310583] Lustre: server umount lustre-OST0000 complete [ 4858.848508] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4858.853255] LustreError: Skipped 2 previous similar messages [ 4862.579932] Lustre: server umount lustre-OST0001 complete [ 4870.708179] Lustre: server umount lustre-MDT0000 complete [ 4872.206667] LustreError: 150858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806849 with bad export cookie 3873694632208618054 [ 4872.209600] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4872.210724] LustreError: 150858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4872.354659] Lustre: server umount lustre-MDT0001 complete [ 4883.805394] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:47:40 (1788806860) [ 4884.772807] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4885.735555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4886.777983] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:47:43 (1788806863) [ 4890.499546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.664741] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4890.679830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4892.826368] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4897.910438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.975859] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4897.988973] Lustre: 157280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4900.378686] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4904.957162] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4908.184652] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4911.834825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4911.956575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4914.909213] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4919.234420] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4919.874855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4921.524233] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4921.641390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4924.225281] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4924.362301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4930.017173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4930.019481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.022539] LustreError: Skipped 2 previous similar messages [ 4930.032180] Lustre: Skipped 5 previous similar messages [ 4931.681898] Lustre: server umount lustre-OST0000 complete [ 4948.283035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4948.506555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.511850] Lustre: Skipped 8 previous similar messages [ 4950.580255] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.591673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.593235] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4950.602277] Lustre: Skipped 1 previous similar message [ 4952.857776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4957.432729] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4960.207406] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4960.375678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4963.105322] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4963.216622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4966.952616] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa065c7fd3000.ost_server_uuid 50 [ 4967.844848] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa065c7fd3000.ost_server_uuid in FULL state after 0 sec [ 4967.945095] LustreError: 157286:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.18@tcp) returned error from blocking AST (req@00000000ec18e672 x1875699209315328 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006fb13490/0x35c220be8921dba0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.18@tcp remote: 0x22672b588f91962 expref: 5 pid: 158463 timeout: 5067 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4967.974244] LustreError: lustre-MDT0000: A client on nid 192.168.201.18@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4967.981283] LustreError: 157274:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.18@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006fb13490/0x35c220be8921dba0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.18@tcp remote: 0x22672b588f91962 expref: 6 pid: 158463 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4970.978473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4970.979820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4970.987531] Lustre: Skipped 15 previous similar messages [ 4975.803490] Lustre: server umount lustre-OST0000 complete [ 4984.280170] Lustre: server umount lustre-MDT0000 complete [ 4986.579226] LustreError: 157269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788806963 with bad export cookie 3873694632208619237 [ 4986.584104] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.587532] LustreError: 157269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4992.832173] Lustre: server umount lustre-MDT0001 complete [ 4998.026983] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4999.617554] Key type lgssc unregistered [ 4999.775342] LNet: 163382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4999.777882] LNetError: 163382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4999.791308] LNet: Removed LNI 192.168.201.118@tcp [ 5000.305309] Key type .llcrypt unregistered [ 5000.307293] Key type ._llcrypt unregistered [ 5013.074540] Key type ._llcrypt registered [ 5013.075784] Key type .llcrypt registered [ 5013.140598] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5013.785588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5013.795893] alg: No test for adler32 (adler32-zlib) [ 5014.734815] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5014.889922] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5016.527349] Key type lgssc registered [ 5017.160876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5021.551864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5021.561510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.810087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5025.188730] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5030.297370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.543759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.770233] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5037.516199] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.873108] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5045.117598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5045.331424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.403420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5048.995742] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5053.546232] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5058.460443] Lustre: Failing over lustre-MDT0000 [ 5058.638492] Lustre: server umount lustre-MDT0000 complete [ 5061.091511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.095458] LustreError: 164763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.097538] Lustre: Skipped 1 previous similar message [ 5061.117023] LustreError: 164763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 5066.210629] LustreError: 164762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5066.235844] LustreError: 164762:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5069.927585] LustreError: 164750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.18@tcp arrived at 1788807047 with bad export cookie 12583213645409872222 [ 5069.935050] LustreError: 164750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5071.330768] LustreError: 165723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.339743] LustreError: 165723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5074.713373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.789457] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.935038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5074.973611] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5077.405515] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5078.641124] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5078.645425] Lustre: lustre-MDT0000: Denying connection for new client a508f673-a2f6-4a4f-b53a-eff35d1402c4 (at 192.168.201.18@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5080.039625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5086.689423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5086.694197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.698459] Lustre: Skipped 1 previous similar message [ 5086.701682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.277237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.281857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.301195] Lustre: Skipped 1 previous similar message [ 5092.072852] Lustre: server umount lustre-OST0000 complete [ 5094.022094] LustreError: 168506:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5094.025903] Lustre: 167856:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5094.032539] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5094.045839] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5094.056075] LustreError: 167856:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000efa0cab8 x1875700080481024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5094.064815] LustreError: 167856:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5094.069910] LustreError: 167856:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5094.078515] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 0 recovered and 2 were evicted. [ 5094.230377] Lustre: server umount lustre-MDT0000 complete [ 5095.395872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.402285] LustreError: 164762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.109079] LustreError: 164750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807073 with bad export cookie 12583213645409872327 [ 5096.109938] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5096.113542] LustreError: 164750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5096.116159] LustreError: 164077:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5102.354138] Lustre: server umount lustre-MDT0001 complete [ 5106.859082] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5108.258358] Key type lgssc unregistered [ 5108.407383] LNet: 169256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5108.413410] LNetError: 169256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5108.424682] LNet: Removed LNI 192.168.201.118@tcp [ 5108.854636] Key type .llcrypt unregistered [ 5108.856066] Key type ._llcrypt unregistered [ 5121.068557] Key type ._llcrypt registered [ 5121.071360] Key type .llcrypt registered [ 5121.138822] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5121.741545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5121.790982] alg: No test for adler32 (adler32-zlib) [ 5122.717109] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5122.897147] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5124.527176] Key type lgssc registered [ 5125.213086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5129.080087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5129.089995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.314542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5132.020867] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5136.487410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.694561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.598341] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5142.140522] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5144.370993] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5147.296111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.425764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.962401] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5153.158079] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.625573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5166.052058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5166.056173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.064242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5167.077942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5167.084785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5167.094331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.196412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.200991] Lustre: Skipped 2 previous similar messages [ 5176.288669] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5176.479818] Lustre: server umount lustre-OST0000 complete [ 5182.435193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.446117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5182.450674] Lustre: Skipped 1 previous similar message [ 5182.457041] Lustre: Skipped 1 previous similar message [ 5186.919581] Lustre: server umount lustre-MDT0000 complete [ 5187.552636] LustreError: 170639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5187.563835] LustreError: 170639:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5191.412819] LustreError: 170623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807168 with bad export cookie 4114995719816696468 [ 5191.417531] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5191.430826] LustreError: 170623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5192.010363] Lustre: server umount lustre-MDT0001 complete [ 5206.596605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5210.609469] Key type lgssc unregistered [ 5211.022805] LNet: 174174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5211.033449] LNetError: 174174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5211.071167] LNet: Removed LNI 192.168.201.118@tcp [ 5212.476159] Key type .llcrypt unregistered [ 5212.479831] Key type ._llcrypt unregistered [ 5244.458979] Key type ._llcrypt registered [ 5244.460516] Key type .llcrypt registered [ 5244.521899] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5245.386336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5245.409954] alg: No test for adler32 (adler32-zlib) [ 5246.471969] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5246.680809] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5248.311238] Key type lgssc registered [ 5250.199619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5265.112736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5265.142074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.062979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.139958] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5277.669413] LustreError: 175854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.685299] LustreError: 175854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5282.803196] LustreError: 175855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.064018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.447100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5288.670133] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5296.138460] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5301.818879] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5307.941796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5308.280397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.393396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5314.926041] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5321.813478] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5327.843514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5327.850267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.861287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5327.869770] Lustre: Skipped 1 previous similar message [ 5327.879980] Lustre: Skipped 1 previous similar message [ 5329.183946] Lustre: server umount lustre-OST0000 complete [ 5332.962332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.964985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5332.973221] Lustre: Skipped 1 previous similar message [ 5332.985149] Lustre: Skipped 1 previous similar message [ 5338.080567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5338.091315] Lustre: Skipped 1 previous similar message [ 5338.968234] Lustre: server umount lustre-MDT0000 complete [ 5342.126761] LustreError: 175835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807319 with bad export cookie 10690868164913988160 [ 5342.129248] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.149054] LustreError: 175835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5342.472182] Lustre: server umount lustre-MDT0001 complete [ 5353.106026] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5355.711187] Key type lgssc unregistered [ 5356.044891] LNet: 179384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5356.054480] LNetError: 179384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5356.086105] LNet: Removed LNI 192.168.201.118@tcp [ 5356.879524] Key type .llcrypt unregistered [ 5356.883179] Key type ._llcrypt unregistered [ 5374.117689] Key type ._llcrypt registered [ 5374.119393] Key type .llcrypt registered [ 5374.217601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5375.221121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5375.267547] alg: No test for adler32 (adler32-zlib) [ 5376.240128] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5376.409796] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5378.024364] Key type lgssc registered [ 5379.137237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5387.643191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5387.663486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.058239] Lustre: *** cfs_fail_loc=135, val=0*** [ 5389.140885] LustreError: 180884:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5389.147666] LustreError: 180884:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -2 [ 5389.168454] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5389.168454] [ 5389.179420] LustreError: MGC192.168.201.118@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5389.196316] LustreError: 180845:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5389.201428] LustreError: 180845:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5389.205209] LustreError: 180845:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5389.210403] LustreError: 180845:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5389.275702] Lustre: server umount lustre-MDT0000 complete [ 5389.279174] LustreError: 180845:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5395.234340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.605949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.656838] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5405.169251] LustreError: 181257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.195854] LustreError: 181257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5407.095349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.415849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5411.667984] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5418.792953] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.370302] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5428.193753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5428.200923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.210117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.241911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.244189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.360260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5432.370384] Lustre: Skipped 2 previous similar messages [ 5433.639745] Lustre: server umount lustre-MDT0000 complete [ 5437.242216] LustreError: 181239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807414 with bad export cookie 2850586954600785874 [ 5437.243409] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.255435] LustreError: 181239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5437.594974] Lustre: server umount lustre-MDT0001 complete [ 5447.586420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5450.390722] Key type lgssc unregistered [ 5450.683850] LNet: 183578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5450.689967] LNetError: 183578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5451.756075] LNet: Removed LNI 192.168.201.118@tcp [ 5452.697550] Key type .llcrypt unregistered [ 5452.699820] Key type ._llcrypt unregistered [ 5474.084164] Key type ._llcrypt registered [ 5474.086462] Key type .llcrypt registered [ 5474.213791] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5474.471488] Key type .llcrypt unregistered [ 5474.480462] Key type ._llcrypt unregistered [ 5488.226624] Key type ._llcrypt registered [ 5488.228868] Key type .llcrypt registered [ 5488.319950] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5489.511098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5489.548106] alg: No test for adler32 (adler32-zlib) [ 5490.548533] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5490.813554] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5492.503288] Key type lgssc registered [ 5493.438724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5499.803703] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5499.821179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5515.299452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5520.415334] LustreError: 185680:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 5547.103211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.814355] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5562.143144] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5568.074124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.548205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.635537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5573.368431] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5573.606957] LustreError: 185706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.629880] LustreError: 185706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5578.726912] LustreError: 185707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.798983] LustreError: 186718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.819073] LustreError: 186718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5582.449177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.726083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5588.190931] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5597.084482] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5604.391370] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5607.567570] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5623.786915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5623.793239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.794334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5623.825146] Lustre: Skipped 1 previous similar message [ 5626.098670] Lustre: server umount lustre-OST0000 complete [ 5634.017255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5634.023272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.028314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.036797] Lustre: Skipped 1 previous similar message [ 5634.043889] Lustre: Skipped 1 previous similar message [ 5636.728233] Lustre: server umount lustre-MDT0000 complete [ 5639.139371] LustreError: 186717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.166938] LustreError: 186717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5640.816943] LustreError: 185688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807617 with bad export cookie 8785961943355849568 [ 5640.830890] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.834553] LustreError: 185688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5641.340916] Lustre: server umount lustre-MDT0001 complete [ 5652.719505] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5656.205604] Key type lgssc unregistered [ 5656.611306] LNet: 189691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.625444] LNetError: 189691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5656.650360] LNet: Removed LNI 192.168.201.118@tcp [ 5657.794784] Key type .llcrypt unregistered [ 5657.801032] Key type ._llcrypt unregistered [ 5685.304518] Key type ._llcrypt registered [ 5685.308190] Key type .llcrypt registered [ 5685.433445] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5686.620833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5686.764968] alg: No test for adler32 (adler32-zlib) [ 5688.196021] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5688.474795] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5690.247311] Key type lgssc registered [ 5691.695395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5700.748132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5700.761971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.222390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5707.873815] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5712.361777] LustreError: 191116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.389526] LustreError: 191116:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5717.279249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.570206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5722.099874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5730.425290] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5737.388817] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5744.760387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.201272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.255440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5752.328523] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5760.918621] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5766.599931] Lustre: Failing over lustre-MDT0000 [ 5766.962554] Lustre: server umount lustre-MDT0000 complete [ 5767.652071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.654889] LustreError: 191112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5767.668588] Lustre: Skipped 1 previous similar message [ 5767.686973] LustreError: 191112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5772.769117] LustreError: 191843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.787234] LustreError: 191843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5777.891471] LustreError: 193057:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.906282] LustreError: 193057:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5782.944158] Lustre: 190425:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788807744/real 1788807744] req@00000000d5bcd715 x1875700786160640/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788807760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5782.984187] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5783.003348] LustreError: 191116:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.033601] LustreError: 191116:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5787.501225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.225048] LustreError: 191115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.262152] LustreError: 191115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5793.618478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.697517] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5798.533325] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5798.621743] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5798.905742] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5798.952574] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5798.995221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5803.404283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5812.006157] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5824.996033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5825.005243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.015026] LustreError: Skipped 1 previous similar message [ 5825.029497] Lustre: Skipped 1 previous similar message [ 5825.036737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5826.047830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5826.067552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.088948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5829.618201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5834.730397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5834.742729] Lustre: Skipped 1 previous similar message [ 5839.327313] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5839.633702] Lustre: server umount lustre-OST0000 complete [ 5844.969971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5844.980051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.984944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.984951] Lustre: Skipped 1 previous similar message [ 5845.021928] Lustre: Skipped 1 previous similar message [ 5850.083794] LustreError: 191843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5850.095595] LustreError: 191843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5850.157153] Lustre: server umount lustre-MDT0000 complete [ 5853.925035] LustreError: 191095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807831 with bad export cookie 16775810455472677684 [ 5853.929644] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5853.937948] LustreError: 191095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5854.257824] Lustre: server umount lustre-MDT0001 complete [ 5865.827672] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5868.472144] Key type lgssc unregistered [ 5868.770736] LNet: 196022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5868.778445] LNetError: 196022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5868.805719] LNet: Removed LNI 192.168.201.118@tcp [ 5869.766211] Key type .llcrypt unregistered [ 5869.771526] Key type ._llcrypt unregistered [ 5894.641266] Key type ._llcrypt registered [ 5894.642409] Key type .llcrypt registered [ 5894.734829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5896.101130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.169745] alg: No test for adler32 (adler32-zlib) [ 5897.246856] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 5897.449829] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5899.191290] Key type lgssc registered [ 5900.358741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5908.944950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5908.980616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.503079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.458498] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5920.742442] LustreError: 197432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5920.758673] LustreError: 197432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5924.394990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.628374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5929.011975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5936.674483] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5942.628760] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5949.610388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5950.098678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.329119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5955.855748] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5962.904257] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5965.270475] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5971.839831] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5989.066483] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5995.490220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5995.499890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5995.501336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5995.506063] Lustre: Skipped 1 previous similar message [ 5995.524874] Lustre: Skipped 1 previous similar message [ 6000.514501] Lustre: server umount lustre-OST0000 complete [ 6005.733879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.749153] Lustre: Skipped 1 previous similar message [ 6005.749845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.770375] Lustre: Skipped 1 previous similar message [ 6009.922138] Lustre: server umount lustre-MDT0000 complete [ 6010.851163] LustreError: 197429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.863867] LustreError: 197429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6013.037136] LustreError: 197413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788807990 with bad export cookie 3702798250405657488 [ 6013.045799] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.050172] LustreError: 197413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6013.304075] Lustre: server umount lustre-MDT0001 complete [ 6023.942979] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6027.194354] Key type lgssc unregistered [ 6027.403717] LNet: 201296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6027.408426] LNetError: 201296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6028.453928] LNet: Removed LNI 192.168.201.118@tcp [ 6029.150569] Key type .llcrypt unregistered [ 6029.153138] Key type ._llcrypt unregistered [ 6048.177241] Key type ._llcrypt registered [ 6048.180974] Key type .llcrypt registered [ 6048.270143] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6049.504624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6049.574533] alg: No test for adler32 (adler32-zlib) [ 6050.629842] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6050.876343] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6052.511272] Key type lgssc registered [ 6053.901849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.558385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.573055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.001901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.280193] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6073.315285] LustreError: 202558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.345659] LustreError: 202558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6075.043724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.337065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.756975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6086.534894] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6091.726272] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6097.673338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6098.012514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6104.042617] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6108.275774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6111.833336] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6117.806922] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6127.957900] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6133.908205] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6142.462455] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6154.727266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6154.727971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6154.749079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6154.761923] Lustre: Skipped 1 previous similar message [ 6159.593888] Lustre: server umount lustre-OST0000 complete [ 6164.962106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.965354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6164.980405] Lustre: Skipped 2 previous similar messages [ 6164.991909] Lustre: Skipped 1 previous similar message [ 6169.899695] Lustre: server umount lustre-MDT0000 complete [ 6170.087410] LustreError: 202558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.835235] LustreError: 202538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808150 with bad export cookie 10425113139192027251 [ 6173.837034] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.843213] LustreError: 202538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6174.290281] Lustre: server umount lustre-MDT0001 complete [ 6185.119949] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6187.476443] Key type lgssc unregistered [ 6187.751533] LNet: 207527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6187.756522] LNetError: 207527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6187.775558] LNet: Removed LNI 192.168.201.118@tcp [ 6188.681497] Key type .llcrypt unregistered [ 6188.685883] Key type ._llcrypt unregistered [ 6208.853766] Key type ._llcrypt registered [ 6208.855480] Key type .llcrypt registered [ 6208.936359] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6209.825879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6209.880409] alg: No test for adler32 (adler32-zlib) [ 6210.947820] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6211.191825] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6212.823171] Key type lgssc registered [ 6213.849399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6220.817052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6220.840989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.387514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6226.239597] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6232.550132] LustreError: 208914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6232.581066] LustreError: 208914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6235.407334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.796923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.313658] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6247.900582] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6253.070681] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6259.722473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6260.025647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.134291] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6267.210556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6273.025371] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6286.817054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6286.821460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6286.829163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6287.848047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.851572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.255481] Lustre: server umount lustre-OST0000 complete [ 6297.073369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6297.083841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.108103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.110879] Lustre: Skipped 1 previous similar message [ 6301.898818] Lustre: server umount lustre-MDT0000 complete [ 6303.203783] LustreError: 208910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6303.221518] LustreError: 208910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6304.852105] LustreError: 208895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808281 with bad export cookie 826358651196967549 [ 6304.859543] LustreError: 208895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6304.866939] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6305.141916] Lustre: server umount lustre-MDT0001 complete [ 6314.840407] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6317.319152] Key type lgssc unregistered [ 6317.565124] LNet: 212599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6317.570859] LNetError: 212599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6318.631470] LNet: Removed LNI 192.168.201.118@tcp [ 6319.450905] Key type .llcrypt unregistered [ 6319.454197] Key type ._llcrypt unregistered [ 6336.281984] Key type ._llcrypt registered [ 6336.284744] Key type .llcrypt registered [ 6336.435148] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6337.349462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6337.421745] alg: No test for adler32 (adler32-zlib) [ 6338.497844] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6338.692441] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6340.320183] Key type lgssc registered [ 6341.400939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6348.618811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6348.637301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.029922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.937838] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6360.547633] LustreError: 213855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.565343] LustreError: 213855:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6361.512719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.837569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6365.858787] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6372.830642] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6378.894890] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6385.406629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6385.806890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.456275] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6395.070459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6398.911780] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6410.724487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.725405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6410.726870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6410.744289] Lustre: Skipped 1 previous similar message [ 6415.850372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6415.859176] Lustre: Skipped 2 previous similar messages [ 6416.937426] Lustre: server umount lustre-OST0000 complete [ 6420.960814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.978695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6420.983920] Lustre: Skipped 1 previous similar message [ 6426.080866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6426.092685] Lustre: Skipped 1 previous similar message [ 6427.024448] Lustre: server umount lustre-MDT0000 complete [ 6430.428638] LustreError: 213836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808407 with bad export cookie 13193630662950876017 [ 6430.434810] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6430.438826] LustreError: 213836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6430.720775] Lustre: server umount lustre-MDT0001 complete [ 6440.989086] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6444.111627] Key type lgssc unregistered [ 6444.550902] LNet: 217537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6444.566206] LNetError: 217537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6444.576632] LNet: Removed LNI 192.168.201.118@tcp [ 6445.700330] Key type .llcrypt unregistered [ 6445.702493] Key type ._llcrypt unregistered [ 6461.008349] Key type ._llcrypt registered [ 6461.013510] Key type .llcrypt registered [ 6461.147832] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6462.221324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6462.243341] alg: No test for adler32 (adler32-zlib) [ 6463.264267] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6463.503473] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6465.159788] Key type lgssc registered [ 6466.256841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6472.838306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6472.854618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.273555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6478.028438] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6484.455831] LustreError: 218777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.477071] LustreError: 218777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6485.394636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.678813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6489.256494] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6494.838332] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6499.180172] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6504.689990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.071778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.120513] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6516.439229] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6519.412744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6529.300404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6529.401740] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6529.641452] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6534.543298] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6541.020495] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6542.338374] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6553.426252] Lustre: Permanently deactivating lustre-OST0001 [ 6553.434538] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6561.832693] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6567.364934] Lustre: Permanently reactivating lustre-OST0001 [ 6567.393442] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6567.399942] Lustre: Skipped 2 previous similar messages [ 6572.575294] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.595146] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6572.602266] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6572.609757] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6572.633145] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6577.646078] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6577.658545] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.679247] Lustre: Skipped 1 previous similar message [ 6577.687135] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6578.146666] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.173341] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6579.175256] Lustre: Skipped 1 previous similar message [ 6581.304287] Lustre: server umount lustre-OST0001 complete [ 6584.306296] LustreError: 221271:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.328199] LustreError: 221271:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6586.338964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6586.345317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6586.361221] Lustre: Skipped 1 previous similar message [ 6586.376498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6589.413204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6589.416401] LustreError: 221272:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.421957] Lustre: Skipped 2 previous similar messages [ 6589.451515] LustreError: 221272:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6590.612610] Lustre: server umount lustre-OST0000 complete [ 6594.545982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6594.549641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6594.569208] Lustre: Skipped 2 previous similar messages [ 6604.774170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6604.778511] Lustre: Skipped 4 previous similar messages [ 6608.351134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6608.546295] Lustre: server umount lustre-MDT0000 complete [ 6609.889994] LustreError: 218777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.904246] LustreError: 218777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6611.701662] LustreError: 218758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808588 with bad export cookie 12025145861065121472 [ 6611.707763] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6611.711663] LustreError: 218758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6612.044745] Lustre: server umount lustre-MDT0001 complete [ 6620.753718] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6622.747569] Key type lgssc unregistered [ 6623.004239] LNet: 223830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6623.007339] LNetError: 223830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6623.024212] LNet: Removed LNI 192.168.201.118@tcp [ 6623.734224] Key type .llcrypt unregistered [ 6623.736921] Key type ._llcrypt unregistered [ 6651.196108] Key type ._llcrypt registered [ 6651.197860] Key type .llcrypt registered [ 6651.273309] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6652.327605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6652.367318] alg: No test for adler32 (adler32-zlib) [ 6653.418134] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6653.725953] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6655.383212] Key type lgssc registered [ 6656.240527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6662.401990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6662.428496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.691727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6663.705170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6663.926363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.224972] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6673.860825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.927806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6673.962772] Lustre: 226009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6674.156832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6677.474727] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6683.925673] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6688.754488] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6694.315715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6694.529534] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6694.765366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6699.818069] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6702.722448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6705.516406] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6708.993667] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6712.171487] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6720.557438] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6726.890890] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6739.663340] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6739.668666] Lustre: Skipped 1 previous similar message [ 6757.771888] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6757.776764] Lustre: Skipped 1 previous similar message [ 6797.261818] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6797.268595] Lustre: Skipped 4 previous similar messages [ 6872.072526] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6872.077976] Lustre: Skipped 8 previous similar messages [ 6881.760213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6881.769159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6881.795364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6882.276361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6882.283411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.456663] Lustre: server umount lustre-OST0000 complete [ 6887.404104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.406405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6887.419458] Lustre: Skipped 1 previous similar message [ 6887.438503] Lustre: Skipped 3 previous similar messages [ 6892.512690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6892.519962] Lustre: Skipped 1 previous similar message [ 6892.941858] Lustre: server umount lustre-MDT0000 complete [ 6896.060384] LustreError: 228575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808873 with bad export cookie 2252766367764958943 [ 6896.061657] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6896.067261] LustreError: 228575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6896.308282] Lustre: server umount lustre-MDT0001 complete [ 6904.443976] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6906.830410] Key type lgssc unregistered [ 6907.013919] LNet: 230819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6907.021885] LNetError: 230819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6908.069622] LNet: Removed LNI 192.168.201.118@tcp [ 6908.748570] Key type .llcrypt unregistered [ 6908.750285] Key type ._llcrypt unregistered [ 6925.637085] Key type ._llcrypt registered [ 6925.643981] Key type .llcrypt registered [ 6925.733746] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6926.486239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6926.523087] alg: No test for adler32 (adler32-zlib) [ 6927.530299] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 6927.746684] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6929.391177] Key type lgssc registered [ 6930.254935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6936.476516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6936.500284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.783310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.602510] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6947.300626] LustreError: 232208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.333386] LustreError: 232208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6948.392424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.677947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6952.705560] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6959.160585] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6964.029935] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6970.359407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.717354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.136639] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6982.006826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6982.203496] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6997.477340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6997.483966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6997.492536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6997.501131] Lustre: Skipped 1 previous similar message [ 6997.515339] Lustre: Skipped 1 previous similar message [ 7001.684309] Lustre: server umount lustre-OST0000 complete [ 7005.154530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7005.164160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.178558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7007.720291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7007.727629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7010.954080] Lustre: server umount lustre-MDT0000 complete [ 7012.834111] LustreError: 233264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.866760] LustreError: 233264:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7013.975100] LustreError: 232189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788808991 with bad export cookie 4302680062350160942 [ 7013.976502] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7013.993701] LustreError: 232189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7014.357141] Lustre: server umount lustre-MDT0001 complete [ 7023.742479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7026.026519] Key type lgssc unregistered [ 7026.274378] LNet: 235895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7026.284896] LNetError: 235895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7026.300367] LNet: Removed LNI 192.168.201.118@tcp [ 7026.932800] Key type .llcrypt unregistered [ 7026.934761] Key type ._llcrypt unregistered [ 7046.471312] Key type ._llcrypt registered [ 7046.475200] Key type .llcrypt registered [ 7046.560530] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7047.524515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7047.677925] alg: No test for adler32 (adler32-zlib) [ 7048.697833] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 7048.913692] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7050.575219] Key type lgssc registered [ 7051.567349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7321.345580] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7323.518877] Key type lgssc unregistered [ 7323.788838] LNet: 237894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7323.793636] LNetError: 237894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7323.803103] LNet: Removed LNI 192.168.201.118@tcp [ 7324.437889] Key type .llcrypt unregistered [ 7324.440499] Key type ._llcrypt unregistered [ 7345.883942] Key type ._llcrypt registered [ 7345.886901] Key type .llcrypt registered [ 7345.979545] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 7359.745896] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7360.724054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7360.892589] alg: No test for adler32 (adler32-zlib) [ 7361.964640] Lustre: Lustre: Build Version: 2.17.58_39_gb44c962 [ 7362.197994] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7363.879688] Key type lgssc registered [ 7364.763993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7389.367346] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7127 sec ========== 15:29:25 (1788809365) [ 7390.589756] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:29:27 (1788809367) === [ 7393.417737] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:29:29 (1788809369) === [ 7425.350271] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7427.323532] Key type lgssc unregistered [ 7427.541447] LNet: 243122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7427.548419] LNetError: 243122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7427.571431] LNet: Removed LNI 192.168.201.118@tcp [ 7428.252524] Key type .llcrypt unregistered [ 7428.256650] Key type ._llcrypt unregistered