[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 555993303 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003569] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.010000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011023] pid_max: default: 32768 minimum: 301 [ 0.013167] LSM: Security Framework initializing [ 0.015075] Yama: becoming mindful. [ 0.016060] SELinux: Initializing. [ 0.017133] *** VALIDATE selinux *** [ 0.028528] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032864] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033189] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035180] *** VALIDATE tmpfs *** [ 0.037420] *** VALIDATE proc *** [ 0.039065] *** VALIDATE cgroup *** [ 0.040013] *** VALIDATE cgroup2 *** [ 0.042229] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.047045] Spectre V2 : User space: Vulnerable [ 0.048011] Speculative Store Bypass: Vulnerable [ 0.052485] debug: unmapping init [mem 0xffffffffafc59000-0xffffffffafc60fff] [ 0.055091] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056900] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057035] ... version: 2 [ 0.058015] ... bit width: 48 [ 0.059036] ... generic registers: 4 [ 0.060026] ... value mask: 0000ffffffffffff [ 0.061023] ... max period: 00007fffffffffff [ 0.062031] ... fixed-purpose events: 3 [ 0.063014] ... event mask: 000000070000000f [ 0.064550] rcu: Hierarchical SRCU implementation. [ 0.067158] smp: Bringing up secondary CPUs ... [ 0.068845] x86: Booting SMP configuration: [ 0.069060] .... node #0, CPUs: #1 #2 #3 [ 0.079084] smp: Brought up 1 node, 4 CPUs [ 0.081023] smpboot: Max logical packages: 1 [ 0.082027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144000] node 0 deferred pages initialised in 56ms [ 0.147252] devtmpfs: initialized [ 0.148757] x86/mm: Memory block size: 128MB [ 0.154365] gcov: version magic: 0x41383552 [ 0.157233] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161389] pinctrl core: initialized pinctrl subsystem [ 0.164594] [ 0.165000] ************************************************************* [ 0.165000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167022] ** ** [ 0.170021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172013] ** ** [ 0.175016] ** This means that this kernel is built to expose internal ** [ 0.177013] ** IOMMU data structures, which may compromise security on ** [ 0.180017] ** your system. ** [ 0.182015] ** ** [ 0.184146] ** If you see this message and you are not debugging the ** [ 0.187015] ** kernel, report this immediately to your vendor! ** [ 0.190015] ** ** [ 0.192020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195025] ************************************************************* [ 0.199460] NET: Registered protocol family 16 [ 0.201579] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210283] cpuidle: using governor menu [ 0.212904] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.216049] PCI: Using configuration type 1 for base access [ 0.218184] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228704] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231161] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235084] cryptd: max_cpu_qlen set to 1000 [ 0.239157] ACPI: Added _OSI(Module Device) [ 0.240169] ACPI: Added _OSI(Processor Device) [ 0.242020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.249123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258259] ACPI: Interpreter enabled [ 0.260072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262015] ACPI: Using IOAPIC for interrupt routing [ 0.263000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265678] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279719] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283085] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.289108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295039] acpiphp: Slot [2] registered [ 0.297266] acpiphp: Slot [3] registered [ 0.298000] acpiphp: Slot [4] registered [ 0.298157] acpiphp: Slot [5] registered [ 0.299000] acpiphp: Slot [6] registered [ 0.301204] acpiphp: Slot [7] registered [ 0.302168] acpiphp: Slot [8] registered [ 0.304149] acpiphp: Slot [9] registered [ 0.305161] acpiphp: Slot [10] registered [ 0.307286] acpiphp: Slot [11] registered [ 0.309142] acpiphp: Slot [12] registered [ 0.310150] acpiphp: Slot [13] registered [ 0.312147] acpiphp: Slot [14] registered [ 0.315165] acpiphp: Slot [15] registered [ 0.316220] acpiphp: Slot [16] registered [ 0.318169] acpiphp: Slot [17] registered [ 0.320329] acpiphp: Slot [18] registered [ 0.322138] acpiphp: Slot [19] registered [ 0.324156] acpiphp: Slot [20] registered [ 0.326177] acpiphp: Slot [21] registered [ 0.327341] acpiphp: Slot [22] registered [ 0.329150] acpiphp: Slot [23] registered [ 0.331209] acpiphp: Slot [24] registered [ 0.332327] acpiphp: Slot [25] registered [ 0.334318] acpiphp: Slot [26] registered [ 0.336164] acpiphp: Slot [27] registered [ 0.338161] acpiphp: Slot [28] registered [ 0.340133] acpiphp: Slot [29] registered [ 0.341187] acpiphp: Slot [30] registered [ 0.343292] acpiphp: Slot [31] registered [ 0.345105] PCI host bridge to bus 0000:00 [ 0.346025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.349041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354035] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.357039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360377] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.366438] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372644] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.385016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.394029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.398026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.400025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406910] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.409056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.412955] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.417000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.431022] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.437000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.445000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.480027] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.491575] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.501027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.530021] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.539017] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.572020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.583000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.590028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.605023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.643026] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.668819] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.684023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.708021] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.724713] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.734028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.745034] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.770031] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.790000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.791507] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.796526] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.798383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.800425] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.808064] iommu: Default domain type: Passthrough [ 0.809723] SCSI subsystem initialized [ 0.810183] ACPI: bus type USB registered [ 0.812139] usbcore: registered new interface driver usbfs [ 0.814119] usbcore: registered new interface driver hub [ 0.816160] usbcore: registered new device driver usb [ 0.818209] pps_core: LinuxPPS API ver. 1 registered [ 0.820011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.823085] PTP clock support registered [ 0.826412] EDAC MC: Ver: 3.0.0 [ 0.828834] PCI: Using ACPI for IRQ routing [ 0.830000] NetLabel: Initializing [ 0.832016] NetLabel: domain hash size = 128 [ 0.834020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.836585] NetLabel: unlabeled traffic allowed by default [ 0.839004] vgaarb: loaded [ 0.841322] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.843016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.854664] clocksource: Switched to clocksource kvm-clock [ 1.014230] VFS: Disk quotas dquot_6.6.0 [ 1.016733] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.019721] *** VALIDATE ramfs *** [ 1.021244] *** VALIDATE hugetlbfs *** [ 1.022828] pnp: PnP ACPI init [ 1.025226] pnp: PnP ACPI: found 6 devices [ 1.043175] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.049214] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.052471] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.054714] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.057325] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.059600] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.062371] NET: Registered protocol family 2 [ 1.065054] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.070568] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.074182] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.079315] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.082866] TCP: Hash tables configured (established 65536 bind 65536) [ 1.085725] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.088837] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.091672] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.094653] NET: Registered protocol family 1 [ 1.099498] RPC: Registered named UNIX socket transport module. [ 1.103305] RPC: Registered udp transport module. [ 1.104932] RPC: Registered tcp transport module. [ 1.110130] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.114572] NET: Registered protocol family 44 [ 1.116777] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.119049] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.121111] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.123383] PCI: CLS 0 bytes, default 64 [ 1.125820] Unpacking initramfs... [ 4.659533] debug: unmapping init [mem 0xffff99e4bcc54000-0xffff99e4bffbffff] [ 4.671373] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.677374] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.686671] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.063199] Initialise system trusted keyrings [ 6.065709] Key type blacklist registered [ 6.072673] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.087960] zbud: loaded [ 6.091726] *** VALIDATE nfs *** [ 6.093077] *** VALIDATE nfs4 *** [ 6.094835] pstore: using deflate compression [ 6.099065] Platform Keyring initialized [ 6.288836] NET: Registered protocol family 38 [ 6.290743] Key type asymmetric registered [ 6.292856] Asymmetric key parser 'x509' registered [ 6.295793] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.306533] io scheduler mq-deadline registered [ 6.308328] io scheduler kyber registered [ 6.310071] io scheduler bfq registered [ 6.313432] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.319818] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.322904] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.326100] ACPI: Power Button [PWRF] [ 6.476713] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.654724] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.016939] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.188650] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.437926] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.495189] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.551765] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.562105] Non-volatile memory driver v1.3 [ 7.566265] Linux agpgart interface v0.103 [ 7.620768] virtio_blk virtio1: [vda] 133160 512-byte logical blocks (68.2 MB/65.0 MiB) [ 7.630660] vda: detected capacity change from 0 to 68177920 [ 7.684770] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.688724] vdb: detected capacity change from 0 to 1073741824 [ 7.756621] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.768856] vdc: detected capacity change from 0 to 2621440000 [ 7.824989] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.833147] vdd: detected capacity change from 0 to 2621440000 [ 7.873438] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.879103] vde: detected capacity change from 0 to 4294967296 [ 7.916858] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.924599] vdf: detected capacity change from 0 to 4294967296 [ 7.939248] libphy: Fixed MDIO Bus: probed [ 7.948497] usbcore: registered new interface driver usbserial_generic [ 7.952384] usbserial: USB Serial support registered for generic [ 7.958586] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.966632] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.970955] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.979356] mousedev: PS/2 mouse device common for all mice [ 7.987688] rtc_cmos 00:05: RTC can wake from S4 [ 7.999298] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.002502] rtc_cmos 00:05: registered as rtc0 [ 8.015430] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.023354] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.023420] intel_pstate: CPU model not supported [ 8.032728] hid: raw HID events driver (C) Jiri Kosina [ 8.041929] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.057356] usbcore: registered new interface driver usbhid [ 8.061049] usbhid: USB HID core driver [ 8.065505] drop_monitor: Initializing network drop monitor service [ 8.067767] Initializing XFRM netlink socket [ 8.074763] NET: Registered protocol family 10 [ 8.084864] Segment Routing with IPv6 [ 8.089483] NET: Registered protocol family 17 [ 8.096358] mpls_gso: MPLS GSO support [ 8.108426] RAS: Correctable Errors collector initialized. [ 8.111662] AVX version of gcm_enc/dec engaged. [ 8.114850] AES CTR mode by8 optimization enabled [ 8.350758] sched_clock: Marking stable (8350718868, 0)->(9635501643, -1284782775) [ 8.377677] registered taskstats version 1 [ 8.386739] Loading compiled-in X.509 certificates [ 8.390969] zswap: loaded using pool lzo/zbud [ 8.484406] Key type big_key registered [ 8.505473] Key type encrypted registered [ 8.507129] ima: No TPM chip found, activating TPM-bypass! [ 8.508920] ima: Allocated hash algorithm: sha1 [ 8.510504] ima: No architecture policies found [ 8.512102] evm: Initialising EVM extended attributes: [ 8.523824] evm: security.selinux [ 8.525546] evm: security.ima [ 8.526798] evm: security.capability [ 8.528246] evm: HMAC attrs: 0x1 [ 8.531747] rtc_cmos 00:05: setting system clock to 2025-09-24 18:03:18 UTC (1758736998) [ 8.544178] debug: unmapping init [mem 0xffffffffb0c03000-0xffffffffb0dfffff] [ 8.552783] debug: unmapping init [mem 0xffffffffaf982000-0xffffffffafc58fff] [ 8.564418] Write protecting the kernel read-only data: 28672k [ 8.568922] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 8.579448] debug: unmapping init [mem 0xffffffffae914000-0xffffffffae9fffff] [ 8.644655] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.662600] systemd[1]: Detected virtualization kvm. [ 8.666711] systemd[1]: Detected architecture x86-64. [ 8.668507] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.712650] systemd[1]: No hostname configured. [ 8.717255] systemd[1]: Set hostname to . [ 8.724602] random: systemd: uninitialized urandom read (16 bytes read) [ 8.728868] systemd[1]: Initializing machine ID from random generator. [ 9.184735] random: systemd: uninitialized urandom read (16 bytes read) [ 9.187463] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.196228] random: systemd: uninitialized urandom read (16 bytes read) [ 9.201666] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.210601] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.141968] device-mapper: uevent: version 1.0.3 [ 11.148719] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.057878] virtio_net virtio0 ens2: renamed from eth0 [ 13.334572] random: fast init done [ 13.640652] scsi host0: ata_piix [ 13.747900] scsi host1: ata_piix [ 13.754122] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.774385] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.052796] random: crng init done [ 19.054008] random: 7 urandom warning(s) missed due to ratelimiting [ 22.141336] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 23.732088] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.053338] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.914171] SELinux: Disabled at runtime. [ 28.064446] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.120451] systemd[1]: Detected virtualization kvm. [ 28.122332] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.190290] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.203595] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.233604] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.255435] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.270322] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.289760] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.314621] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK [[ 30.834293] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 32.030480] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.536978] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.719987] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.853214] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.380632] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 41.201944] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 42.326240] NFS: Registering the id_resolver key type [ 42.328105] Key type id_resolver registered [ 42.329608] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 44.629018] hrtimer: interrupt took 4005687 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg245-server login: [ 90.950243] libcfs: loading out-of-tree module taints kernel. [ 90.967899] Key type ._llcrypt registered [ 90.969252] Key type .llcrypt registered [ 91.059642] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 104.047396] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 105.176099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.186969] alg: No test for adler32 (adler32-zlib) [ 106.382501] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 106.929188] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 108.623113] Key type lgssc registered [ 109.774778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.680601] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 123.352718] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.630990] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.467811] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.757293] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.724464] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 164.599367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.665489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 164.685877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.856644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.879957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.937726] Lustre: lustre-MDT0000: new disk, initializing [ 166.027209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 166.040117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.977311] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.938876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.994297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 179.043117] Lustre: 6516:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.067395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.070931] Lustre: Skipped 1 previous similar message [ 179.134384] Lustre: lustre-MDT0001: new disk, initializing [ 179.188724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.220091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.225457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.109749] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.901082] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.784449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.840703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.987744] Lustre: lustre-OST0000: new disk, initializing [ 192.990943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.035716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.270661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.277968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.355952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 196.717411] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.335639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 207.389828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 207.488772] Lustre: lustre-OST0001: new disk, initializing [ 207.491578] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.551879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.727613] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.625226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.637467] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.686068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.344225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.595573] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.053988] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.876508] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing yml_node [ 242.349099] Lustre: DEBUG MARKER: Client: 2.16.58.53 [ 244.241838] Lustre: DEBUG MARKER: MDS: 2.16.58.53 [ 245.993856] Lustre: DEBUG MARKER: OSS: 2.16.58.53 [ 247.126911] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 24 14:07:16 EDT 2025 [ 260.654504] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 262.014116] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 276.448638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 276.458859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.471514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.471933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.472245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.476959] Lustre: Skipped 1 previous similar message [ 277.489855] Lustre: Skipped 1 previous similar message [ 282.411059] LustreError: 12217:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 282.659687] Lustre: server umount lustre-MDT0000 complete [ 285.592279] LustreError: 6508:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737275 with bad export cookie 12874770664409765151 [ 285.600406] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.606934] LustreError: 6508:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 287.720642] LustreError: 9293:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.727831] LustreError: 9293:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 287.732207] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.736661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.746762] Lustre: Skipped 2 previous similar messages [ 287.750419] Lustre: Skipped 2 previous similar messages [ 290.719899] LustreError: 6525:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.720512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.749754] LustreError: 6525:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 290.757699] Lustre: Skipped 1 previous similar message [ 291.825809] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 291.833621] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 292.135608] Lustre: server umount lustre-MDT0001 complete [ 295.686403] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 295.696341] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 295.750174] Lustre: server umount lustre-OST0000 complete [ 298.797573] LustreError: 12821:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 298.804601] LustreError: 12821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 298.945127] Lustre: server umount lustre-OST0001 complete [ 304.240188] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 311.477862] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 325.395938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.295889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 339.429918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.158179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.995297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 356.134844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.645324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 356.721362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 356.841644] Lustre: lustre-MDT0000: new disk, initializing [ 356.937520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.952834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 362.086295] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 373.412101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 373.525020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 373.588210] Lustre: 15009:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 373.646952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 373.654590] Lustre: Skipped 1 previous similar message [ 373.742950] Lustre: lustre-MDT0001: new disk, initializing [ 373.877151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 373.929943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 373.937316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 378.610701] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 387.108812] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 390.645585] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 397.259723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 397.385964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 397.735454] Lustre: lustre-OST0000: new disk, initializing [ 397.748739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 397.840565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.971955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 398.975081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 399.125939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 403.425738] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 410.712270] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 415.054867] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 415.264606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 418.612581] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 418.795535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 424.938503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.942238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.948714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 426.555947] LustreError: 17946:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 426.559274] LustreError: 17946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 426.723777] Lustre: server umount lustre-OST0000 complete [ 435.173833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 435.176404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.186770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 435.186779] Lustre: Skipped 1 previous similar message [ 435.188123] LustreError: Skipped 1 previous similar message [ 435.204655] Lustre: Skipped 2 previous similar messages [ 444.383279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.479601] LustreError: 18148:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 444.486333] LustreError: 18148:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 444.558622] Lustre: server umount lustre-MDT0000 complete [ 445.412717] LustreError: 15836:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.440068] LustreError: 15836:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 448.121884] LustreError: 15001:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737438 with bad export cookie 12874770664409766684 [ 448.126312] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.274639] LustreError: 15001:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 448.827688] Lustre: server umount lustre-MDT0001 complete [ 452.271642] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:10:40 (1758737440) [ 457.981738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.519214] LustreError: 18877:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.602081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.181111] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 463.841700] LustreError: 18877:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.145391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 475.218216] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 481.072221] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 484.991678] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 492.562286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 499.030705] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 504.443813] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 513.206575] LustreError: 21121:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 513.208942] LustreError: 21121:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 513.473795] Lustre: server umount lustre-OST0000 complete [ 518.629452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.632629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.651298] Lustre: Skipped 1 previous similar message [ 518.667054] Lustre: Skipped 3 previous similar messages [ 527.923798] Lustre: server umount lustre-MDT0000 complete [ 528.866962] LustreError: 18877:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.892741] LustreError: 18877:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 529.890680] Lustre: 3659:0:(client.c:2464:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1758737503/real 1758737503] req@ffff99e404de5500 x1844169509334272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1758737519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 531.661878] LustreError: 18857:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737521 with bad export cookie 12874770664409767559 [ 531.684217] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.076257] Lustre: server umount lustre-MDT0001 complete [ 544.241201] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 547.609806] Key type lgssc unregistered [ 547.978295] LNet: 22057:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.991700] LNetError: 22057:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.031254] LNet: Removed LNI 192.168.202.145@tcp [ 550.087175] Key type .llcrypt unregistered [ 550.088963] Key type ._llcrypt unregistered [ 570.026906] Key type ._llcrypt registered [ 570.028996] Key type .llcrypt registered [ 570.111478] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 570.985426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 571.021055] alg: No test for adler32 (adler32-zlib) [ 572.000627] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 572.207224] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 573.863160] Key type lgssc registered [ 574.850634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 581.360939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 581.385615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.826757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.337326] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 592.353520] LustreError: 23429:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.370556] LustreError: 23429:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 594.207743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 594.590322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 598.563245] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 603.811600] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 607.209677] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 613.554659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 613.964336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.086895] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 624.156455] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 627.329154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 639.968547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 639.970960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.988843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 640.437095] LustreError: 26118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 640.803969] Lustre: server umount lustre-OST0000 complete [ 646.113955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 646.121370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.135494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.162285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.165159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 653.281124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.283119] Lustre: Skipped 1 previous similar message [ 658.399914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 658.402643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 658.413537] Lustre: Skipped 1 previous similar message [ 658.484925] LustreError: 26319:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 658.498400] LustreError: 26319:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 658.654180] Lustre: server umount lustre-MDT0000 complete [ 661.654161] LustreError: 23410:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737651 with bad export cookie 10721264645447443862 [ 661.667796] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 661.789194] LustreError: 26521:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 661.792270] LustreError: 26521:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 661.979705] Lustre: server umount lustre-MDT0001 complete [ 670.540186] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 673.131769] Key type lgssc unregistered [ 673.416166] LNet: 27053:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.433054] LNetError: 27053:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.460110] LNet: Removed LNI 192.168.202.145@tcp [ 674.408247] Key type .llcrypt unregistered [ 674.411676] Key type ._llcrypt unregistered [ 694.757749] Key type ._llcrypt registered [ 694.761763] Key type .llcrypt registered [ 694.869809] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 695.669355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 695.757213] alg: No test for adler32 (adler32-zlib) [ 696.933421] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 697.174730] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 698.911188] Key type lgssc registered [ 700.172305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 707.183339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 707.197685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.640211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.794193] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 718.819231] LustreError: 28426:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.839422] LustreError: 28426:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 718.897920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 719.197128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 722.092731] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 726.510372] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.481108] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 741.375916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 741.734565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.769558] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 751.218935] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 754.028379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 760.289490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 760.301268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.316729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 764.391353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 764.391962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 764.399825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.505676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 769.514643] Lustre: Skipped 2 previous similar messages [ 774.111148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 774.142141] LustreError: 31018:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 774.248319] Lustre: server umount lustre-OST0000 complete [ 779.754538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.756178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.776379] Lustre: Skipped 1 previous similar message [ 779.785279] Lustre: Skipped 1 previous similar message [ 783.436348] LustreError: 31221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 783.440426] LustreError: 31221:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 783.515253] Lustre: server umount lustre-MDT0000 complete [ 784.865407] LustreError: 28421:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.877458] LustreError: 28421:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 786.201910] LustreError: 30932:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737776 with bad export cookie 14125814405923804238 [ 786.208441] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.210957] LustreError: 30932:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 786.371689] LustreError: 31424:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 786.378428] LustreError: 31424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 786.568784] Lustre: server umount lustre-MDT0001 complete [ 794.274086] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 796.099422] Key type lgssc unregistered [ 796.291710] LNet: 31955:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.300084] LNetError: 31955:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.310958] LNet: Removed LNI 192.168.202.145@tcp [ 796.805116] Key type .llcrypt unregistered [ 796.806893] Key type ._llcrypt unregistered [ 813.242359] Key type ._llcrypt registered [ 813.244014] Key type .llcrypt registered [ 813.333963] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 814.094672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.159619] alg: No test for adler32 (adler32-zlib) [ 815.120876] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 815.331568] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 816.959776] Key type lgssc registered [ 817.791382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 824.014183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 824.035500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.484162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.784440] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 835.045274] LustreError: 33329:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.060570] LustreError: 33329:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 835.806743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 836.086799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 839.166428] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 843.454391] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 846.675881] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 851.240211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 851.498399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.118418] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 859.326359] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 870.959359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 881.121615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.132470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 881.137269] Lustre: Skipped 1 previous similar message [ 881.142118] Lustre: Skipped 1 previous similar message [ 886.250797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 886.262370] Lustre: Skipped 1 previous similar message [ 891.359159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 891.366197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 891.384189] Lustre: Skipped 1 previous similar message [ 891.440299] LustreError: 35593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 891.625504] Lustre: server umount lustre-OST0000 complete [ 896.480572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.491931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 896.495627] Lustre: Skipped 1 previous similar message [ 896.508175] Lustre: Skipped 1 previous similar message [ 901.601192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.604023] Lustre: Skipped 1 previous similar message [ 909.281089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 909.346261] LustreError: 35798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 909.348734] LustreError: 35798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 909.420667] Lustre: server umount lustre-MDT0000 complete [ 911.848254] LustreError: 35595:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 911.871183] LustreError: 35595:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 912.163441] LustreError: 33309:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758737902 with bad export cookie 7699672771694813723 [ 912.168965] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.170289] LustreError: 33309:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 912.323099] LustreError: 36003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 912.330262] LustreError: 36003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 912.504871] Lustre: server umount lustre-MDT0001 complete [ 920.974721] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 923.544797] Key type lgssc unregistered [ 923.859956] LNet: 36535:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.868044] LNetError: 36535:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.887388] LNet: Removed LNI 192.168.202.145@tcp [ 924.731586] Key type .llcrypt unregistered [ 924.733604] Key type ._llcrypt unregistered [ 942.944329] Key type ._llcrypt registered [ 942.946567] Key type .llcrypt registered [ 943.090771] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 944.008681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.025441] alg: No test for adler32 (adler32-zlib) [ 945.064952] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 945.262470] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 946.931108] Key type lgssc registered [ 947.972131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.561176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 954.589283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.926256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.146131] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 965.097260] LustreError: 37907:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.111970] LustreError: 37907:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 966.138884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 966.401359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 969.429900] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 973.152714] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 975.602424] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 981.270420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 981.595232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.057976] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 989.733217] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.066238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1010.820819] Lustre: lustre-OST0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1011.681402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1011.688132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.702443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1011.719678] Lustre: Skipped 1 previous similar message [ 1015.908605] LustreError: 40177:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1015.932211] LustreError: 39506:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.065421] Lustre: server umount lustre-OST0000 complete [ 1036.261171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.267740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.282854] Lustre: Skipped 1 previous similar message [ 1036.285810] Lustre: Skipped 1 previous similar message [ 1040.007699] LustreError: 40552:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1040.015910] LustreError: 40552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1040.169493] Lustre: server umount lustre-MDT0000 complete [ 1041.391630] LustreError: 37908:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.411139] LustreError: 37908:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1044.096340] LustreError: 37889:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738034 with bad export cookie 15878129504917958347 [ 1044.100377] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.115596] LustreError: 37889:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1050.370603] LustreError: 40755:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1050.377386] LustreError: 40755:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1050.656987] Lustre: server umount lustre-MDT0001 complete [ 1061.487244] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1064.555590] Key type lgssc unregistered [ 1064.894802] LNet: 41287:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.909078] LNetError: 41287:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.947777] LNet: Removed LNI 192.168.202.145@tcp [ 1065.850479] Key type .llcrypt unregistered [ 1065.852324] Key type ._llcrypt unregistered [ 1088.938692] Key type ._llcrypt registered [ 1088.944793] Key type .llcrypt registered [ 1089.034277] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1089.827898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1089.869096] alg: No test for adler32 (adler32-zlib) [ 1090.872364] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1091.107395] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1092.823243] Key type lgssc registered [ 1094.188218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1102.644143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1102.682354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.220276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.988989] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1114.600119] LustreError: 42681:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.622624] LustreError: 42681:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1116.585316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1116.957251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.916177] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1125.956141] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1129.483731] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1136.275772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1136.568110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.521988] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1147.905466] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1149.876452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1159.830529] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1160.162199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.182601] Lustre: Skipped 2 previous similar messages [ 1164.905943] LustreError: 44928:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1164.914724] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.45@tcp (stopping) [ 1164.926691] Lustre: Skipped 3 previous similar messages [ 1164.999797] Lustre: server umount lustre-MDT0000 complete [ 1165.281293] LustreError: 43395:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.289485] LustreError: 43395:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1168.035474] LustreError: 42662:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738158 with bad export cookie 10968173937324217693 [ 1168.041430] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.050345] LustreError: 42662:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1170.037483] LustreError: 43710:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.063431] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.45@tcp (stopping) [ 1170.411521] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.409583] LustreError: 42678:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.414132] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1173.435862] LustreError: 42678:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1173.470034] Lustre: Skipped 1 previous similar message [ 1174.244971] LustreError: 45132:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1174.270675] LustreError: 45132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1174.462797] Lustre: server umount lustre-MDT0001 complete [ 1186.414547] LustreError: 42662:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.45@tcp arrived at 1758738176 with bad export cookie 10968173937324218358 [ 1186.444942] LustreError: 42662:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1204.476482] LustreError: 45357:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1204.483936] LustreError: 45357:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1204.657970] Lustre: server umount lustre-OST0000 complete [ 1219.591060] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1222.395842] Key type lgssc unregistered [ 1222.650565] LNet: 46196:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.661662] LNetError: 46196:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.681492] LNet: Removed LNI 192.168.202.145@tcp [ 1223.594499] Key type .llcrypt unregistered [ 1223.597829] Key type ._llcrypt unregistered [ 1242.936620] Key type ._llcrypt registered [ 1242.938269] Key type .llcrypt registered [ 1243.036027] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1243.801123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1243.822501] alg: No test for adler32 (adler32-zlib) [ 1244.793218] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1244.968473] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1246.623185] Key type lgssc registered [ 1247.333558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1252.983917] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1252.996708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1270.559654] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1276.703168] LustreError: 47541:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 1307.487228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.543056] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1315.369544] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1481.053797] LustreError: 48250:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1481.192091] Lustre: server umount lustre-OST0000 complete [ 1493.737876] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1495.970337] Key type lgssc unregistered [ 1496.248598] LNet: 49088:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.256535] LNetError: 49088:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.287360] LNet: Removed LNI 192.168.202.145@tcp [ 1496.968466] Key type .llcrypt unregistered [ 1496.971338] Key type ._llcrypt unregistered [ 1511.548092] Key type ._llcrypt registered [ 1511.549699] Key type .llcrypt registered [ 1511.613959] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1512.251834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1512.270158] alg: No test for adler32 (adler32-zlib) [ 1513.194892] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1513.348606] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1514.967411] Key type lgssc registered [ 1515.865399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.176781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1521.201507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.592409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.850359] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1530.707322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1531.020354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1534.029983] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1537.813581] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1540.196622] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1544.364994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1544.624560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.697135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:257) [ 1548.602300] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1552.489276] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1555.367389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.387795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.370133] LustreError: 52704:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1560.557153] Lustre: server umount lustre-OST0000 complete [ 1565.666966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.669677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.693750] Lustre: Skipped 1 previous similar message [ 1565.699049] Lustre: Skipped 1 previous similar message [ 1570.022503] LustreError: 52905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1570.032333] LustreError: 52905:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1570.249354] Lustre: server umount lustre-MDT0000 complete [ 1570.786298] LustreError: 50462:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.797470] LustreError: 50462:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1573.245979] LustreError: 50443:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738563 with bad export cookie 11003603526959613512 [ 1573.255962] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1573.394272] LustreError: 53106:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1573.397546] LustreError: 53106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1573.610301] Lustre: server umount lustre-MDT0001 complete [ 1581.101895] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1583.107414] Key type lgssc unregistered [ 1583.431612] LNet: 53638:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.447138] LNetError: 53638:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.460568] LNet: Removed LNI 192.168.202.145@tcp [ 1584.133635] Key type .llcrypt unregistered [ 1584.135413] Key type ._llcrypt unregistered [ 1600.487199] Key type ._llcrypt registered [ 1600.488709] Key type .llcrypt registered [ 1600.573179] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1601.238466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.254877] alg: No test for adler32 (adler32-zlib) [ 1602.249927] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1602.400712] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1604.031986] Key type lgssc registered [ 1604.885261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1610.499437] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1610.523529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1627.103600] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1633.247408] LustreError: 54984:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 1663.008615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.659397] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1671.573821] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1676.447513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.820251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.868583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:289) [ 1679.252211] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1682.913872] LustreError: 55011:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.921421] LustreError: 55011:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 1684.863961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1685.120468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1687.881771] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1691.846593] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1694.497656] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1697.251182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1697.254890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.264565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.321318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1700.327624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.342212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.136337] LustreError: 57259:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1702.242421] Lustre: server umount lustre-OST0000 complete [ 1720.800978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1720.809291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.818737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.823080] Lustre: Skipped 1 previous similar message [ 1726.031831] LustreError: 57636:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1726.034510] LustreError: 57636:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1726.125190] Lustre: server umount lustre-MDT0000 complete [ 1727.456585] LustreError: 55914:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.471168] LustreError: 55914:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1729.019734] LustreError: 54992:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738718 with bad export cookie 12056525296074262167 [ 1729.027742] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1743.328057] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1743.451183] LustreError: 57837:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1743.468268] LustreError: 57837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1743.808350] Lustre: server umount lustre-MDT0001 complete [ 1753.735662] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1755.840678] Key type lgssc unregistered [ 1756.069053] LNet: 58372:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.079665] LNetError: 58372:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.089730] LNet: Removed LNI 192.168.202.145@tcp [ 1756.701367] Key type .llcrypt unregistered [ 1756.703545] Key type ._llcrypt unregistered [ 1774.006881] Key type ._llcrypt registered [ 1774.008080] Key type .llcrypt registered [ 1774.203356] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1774.914683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.013440] alg: No test for adler32 (adler32-zlib) [ 1775.996605] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1776.149076] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1777.807210] Key type lgssc registered [ 1778.887520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1784.875846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.891781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.336400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1789.525316] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1796.387863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1796.586792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1796.736336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.779805] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1804.694650] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1808.022602] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1813.596241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1813.921562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.473652] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1823.914313] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1831.353489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:321) [ 1841.638789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.644426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.644730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.647748] Lustre: Skipped 1 previous similar message [ 1844.364674] LustreError: 62014:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1844.531416] Lustre: server umount lustre-OST0000 complete [ 1851.872403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1851.881162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.899841] Lustre: Skipped 1 previous similar message [ 1851.900498] Lustre: Skipped 1 previous similar message [ 1854.388745] LustreError: 62216:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1854.395829] LustreError: 62216:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1854.514295] Lustre: server umount lustre-MDT0000 complete [ 1856.992495] LustreError: 59740:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1857.001968] LustreError: 59740:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1857.357826] LustreError: 61366:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738847 with bad export cookie 6865862052421362668 [ 1857.361658] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1857.366567] LustreError: 61366:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1857.488526] LustreError: 62417:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1857.491941] LustreError: 62417:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1857.648570] Lustre: server umount lustre-MDT0001 complete [ 1866.182889] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 1868.587588] Key type lgssc unregistered [ 1868.867234] LNet: 62947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.879484] LNetError: 62947:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.901700] LNet: Removed LNI 192.168.202.145@tcp [ 1869.703087] Key type .llcrypt unregistered [ 1869.704625] Key type ._llcrypt unregistered [ 1896.416839] Key type ._llcrypt registered [ 1896.418438] Key type .llcrypt registered [ 1896.538481] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 1897.398178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.443036] alg: No test for adler32 (adler32-zlib) [ 1898.489506] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 1898.713687] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 1900.367128] Key type lgssc registered [ 1901.341034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1908.440696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1908.472174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.899226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.558551] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1919.468949] LustreError: 64610:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.485801] LustreError: 64610:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1920.278271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1920.619532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.919926] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1928.456707] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1931.932652] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1938.380762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1938.668857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.494478] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1948.920126] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1955.073378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:353) [ 1963.659319] Lustre: Failing over lustre-MDT0000 [ 1963.834140] LustreError: 66879:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1963.931311] Lustre: server umount lustre-MDT0000 complete [ 1964.676581] LustreError: 66580:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1965.538521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.574715] Lustre: Skipped 2 previous similar messages [ 1969.776945] LustreError: 64605:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.795237] LustreError: 64605:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1972.396804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.542919] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.766142] Lustre: *** cfs_fail_loc=135, val=0*** [ 1972.902881] LustreError: 67388:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1972.906725] LustreError: 67388:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -2 [ 1972.920328] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1972.920328] [ 1972.935479] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1972.967523] LustreError: 67376:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1972.983089] LustreError: 67376:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1972.994748] LustreError: 67376:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1973.042217] Lustre: server umount lustre-MDT0000 complete [ 1973.058087] LustreError: 67376:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1974.908977] LustreError: 66580:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.929490] LustreError: 66580:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1978.952665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.158562] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.407203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.447917] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1980.032067] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1983.149772] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 1984.502225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1984.533914] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1984.610651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:385) [ 1990.118085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.129151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.155993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1991.675139] LustreError: 68198:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1991.689408] LustreError: 68198:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1991.816603] Lustre: server umount lustre-OST0000 complete [ 1999.841635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.846436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1999.851470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.864412] Lustre: Skipped 1 previous similar message [ 2001.598030] LustreError: 68398:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2001.611685] LustreError: 68398:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2001.760764] Lustre: server umount lustre-MDT0000 complete [ 2004.963721] LustreError: 64610:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.995303] LustreError: 64610:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 2005.443051] LustreError: 64590:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758738995 with bad export cookie 4011914594341759515 [ 2005.451602] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.454606] LustreError: 64590:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2015.724732] LustreError: 68600:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2015.727732] LustreError: 68600:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2015.917907] Lustre: server umount lustre-MDT0001 complete [ 2026.152063] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2028.694330] Key type lgssc unregistered [ 2029.009207] LNet: 69134:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.018942] LNetError: 69134:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.053706] LNet: Removed LNI 192.168.202.145@tcp [ 2030.921955] Key type .llcrypt unregistered [ 2030.923882] Key type ._llcrypt unregistered [ 2052.203932] Key type ._llcrypt registered [ 2052.209858] Key type .llcrypt registered [ 2052.301548] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2053.272496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.323416] alg: No test for adler32 (adler32-zlib) [ 2054.395425] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2054.634198] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2056.303811] Key type lgssc registered [ 2057.368271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2063.873679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2063.891825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.342909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.809981] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2075.624921] LustreError: 70528:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.636738] LustreError: 70528:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2076.251299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2076.575132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2079.533934] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2084.091023] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2087.574511] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2094.101938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2094.576102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.243166] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2105.648711] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2110.961512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:417) [ 2116.480810] Lustre: Failing over lustre-MDT0000 [ 2116.645029] LustreError: 72775:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2116.708236] Lustre: server umount lustre-MDT0000 complete [ 2117.605139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2117.614600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.633673] LustreError: 72687:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.187296] LustreError: 70529:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.188372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.210232] LustreError: 70529:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2125.560561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.763312] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.904397] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2126.026686] LustreError: 73285:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2126.041730] LustreError: 73285:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -9 [ 2126.061331] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2126.061331] [ 2126.071290] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2126.098700] LustreError: 73273:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2126.109365] LustreError: 73273:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2126.143689] LustreError: 73273:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2126.147659] LustreError: 70509:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739116 with bad export cookie 8266430276617497291 [ 2126.161747] Lustre: server umount lustre-MDT0000 complete [ 2126.163633] LustreError: 73273:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2126.306359] LustreError: 70529:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.333891] LustreError: 70529:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 2131.428036] LustreError: 70529:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.435387] LustreError: 70529:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2132.554990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.546821] LustreError: 69837:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f5c753f7 x1844171551831424/t0(0) o250->MGC192.168.202.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.945474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.012420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2138.227766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2140.807512] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2142.198730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2142.239106] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2142.290500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:449) [ 2147.814692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2147.817843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.827338] LustreError: Skipped 1 previous similar message [ 2147.828505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2147.849427] Lustre: Skipped 2 previous similar messages [ 2152.420993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.437772] Lustre: Skipped 1 previous similar message [ 2157.542683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2157.548916] Lustre: Skipped 1 previous similar message [ 2162.144420] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2162.181115] LustreError: 74097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2162.183893] LustreError: 74097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2162.385394] Lustre: server umount lustre-OST0000 complete [ 2167.777405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2167.781963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.790065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.790076] Lustre: Skipped 1 previous similar message [ 2167.819159] Lustre: Skipped 1 previous similar message [ 2172.411782] LustreError: 74302:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2172.417526] LustreError: 74302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2172.674434] Lustre: server umount lustre-MDT0000 complete [ 2172.905081] LustreError: 70525:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.924572] LustreError: 70525:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 2176.464108] LustreError: 70509:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739166 with bad export cookie 8266430276617497445 [ 2176.474660] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.488478] LustreError: 70509:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2176.504782] LustreError: Skipped 1 previous similar message [ 2176.763378] LustreError: 74506:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2176.775284] LustreError: 74506:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2177.222383] Lustre: server umount lustre-MDT0001 complete [ 2187.619245] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2191.163976] Key type lgssc unregistered [ 2191.459863] LNet: 75039:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.470500] LNetError: 75039:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.496698] LNet: Removed LNI 192.168.202.145@tcp [ 2192.472092] Key type .llcrypt unregistered [ 2192.474928] Key type ._llcrypt unregistered [ 2204.272878] Key type ._llcrypt registered [ 2204.274187] Key type .llcrypt registered [ 2204.358506] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2204.697451] Key type .llcrypt unregistered [ 2204.703406] Key type ._llcrypt unregistered [ 2226.565277] Key type ._llcrypt registered [ 2226.566581] Key type .llcrypt registered [ 2226.674032] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2227.987424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2228.070751] alg: No test for adler32 (adler32-zlib) [ 2229.139703] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2229.450543] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2231.132574] Key type lgssc registered [ 2232.223715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2240.085944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2240.102746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.812256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.174820] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2251.242030] LustreError: 76900:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.254290] LustreError: 76900:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2255.221285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2255.527129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.233730] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2264.411322] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2267.720671] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2274.402468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2274.829204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2279.736989] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2284.159881] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2286.135740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:227 to 0x280000401:481) [ 2301.412801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2301.414031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.424557] LustreError: Skipped 1 previous similar message [ 2301.425207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.430361] Lustre: Skipped 1 previous similar message [ 2301.455830] Lustre: Skipped 1 previous similar message [ 2303.170196] LustreError: 79167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2303.375775] Lustre: server umount lustre-OST0000 complete [ 2307.044551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2307.048657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.065178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.648527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.651379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.881680] LustreError: 79369:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2312.895849] LustreError: 79369:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2314.981179] LustreError: 79369:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2314.991244] LustreError: 79369:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2315.047733] Lustre: server umount lustre-MDT0000 complete [ 2316.769906] LustreError: 76894:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.794173] LustreError: 76894:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2318.660505] LustreError: 76879:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739308 with bad export cookie 7066450405176447859 [ 2318.671762] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2318.682494] LustreError: 76879:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2318.815618] LustreError: 79570:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2318.824320] LustreError: 79570:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2319.115349] Lustre: server umount lustre-MDT0001 complete [ 2329.461781] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2331.981624] Key type lgssc unregistered [ 2332.288229] LNet: 80103:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.303612] LNetError: 80103:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.326736] LNet: Removed LNI 192.168.202.145@tcp [ 2332.940602] Key type .llcrypt unregistered [ 2332.942898] Key type ._llcrypt unregistered [ 2351.666991] Key type ._llcrypt registered [ 2351.668703] Key type .llcrypt registered [ 2351.791879] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2352.774673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.792587] alg: No test for adler32 (adler32-zlib) [ 2353.778445] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2353.951339] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2355.631876] Key type lgssc registered [ 2356.499266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2362.640807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2362.658083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.967300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2366.427649] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2373.093052] LustreError: 81477:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.108807] LustreError: 81477:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2373.683280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2373.906257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.165823] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2381.841319] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2385.119534] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2391.260867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2391.658331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.904209] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2401.611529] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2409.080361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:483 to 0x280000401:513) [ 2419.682620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.688915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.721302] LustreError: Skipped 1 previous similar message [ 2419.722301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.751754] Lustre: Skipped 1 previous similar message [ 2422.859746] LustreError: 83743:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2423.087322] Lustre: server umount lustre-OST0000 complete [ 2429.925037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.936672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.947445] Lustre: Skipped 2 previous similar messages [ 2432.839876] LustreError: 83945:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2432.851523] LustreError: 83945:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2432.971430] Lustre: server umount lustre-MDT0000 complete [ 2435.041060] LustreError: 82540:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.051072] LustreError: 82540:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2436.046856] LustreError: 83096:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739426 with bad export cookie 867017559823279034 [ 2436.048050] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.068164] LustreError: 83096:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2436.173247] LustreError: 84151:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2436.177112] LustreError: 84151:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2436.382454] Lustre: server umount lustre-MDT0001 complete [ 2446.927561] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2449.630233] Key type lgssc unregistered [ 2450.005797] LNet: 84683:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.028483] LNetError: 84683:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.049170] LNet: Removed LNI 192.168.202.145@tcp [ 2451.962663] Key type .llcrypt unregistered [ 2451.970879] Key type ._llcrypt unregistered [ 2475.870753] Key type ._llcrypt registered [ 2475.873118] Key type .llcrypt registered [ 2475.952367] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2477.438606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.490825] alg: No test for adler32 (adler32-zlib) [ 2478.500770] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2478.751993] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2480.471192] Key type lgssc registered [ 2481.580828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2488.647495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.679082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.240661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.683703] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2500.587042] LustreError: 86077:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.634594] LustreError: 86077:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2502.442916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2502.775276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2506.473488] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2511.518513] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2514.490538] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2520.377918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2520.622842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.198838] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2529.545878] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2536.001642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:483 to 0x280000401:545) [ 2545.543777] LustreError: 88342:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2545.731994] Lustre: server umount lustre-OST0000 complete [ 2551.265401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2551.268367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2551.272669] Lustre: Skipped 1 previous similar message [ 2556.385429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.393105] Lustre: Skipped 2 previous similar messages [ 2558.442711] LustreError: 88543:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2558.457617] LustreError: 88543:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2558.532643] Lustre: server umount lustre-MDT0000 complete [ 2561.505949] LustreError: 86789:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.514969] LustreError: 86789:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2561.647391] LustreError: 86058:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739551 with bad export cookie 14416884619298734464 [ 2561.653738] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.248402] LustreError: 88745:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2571.251726] LustreError: 88745:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2571.482886] Lustre: server umount lustre-MDT0001 complete [ 2582.432472] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2585.802445] Key type lgssc unregistered [ 2586.165660] LNet: 89278:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.171965] LNetError: 89278:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2586.182537] LNet: Removed LNI 192.168.202.145@tcp [ 2587.074216] Key type .llcrypt unregistered [ 2587.078756] Key type ._llcrypt unregistered [ 2608.436369] Key type ._llcrypt registered [ 2608.438183] Key type .llcrypt registered [ 2608.544987] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2609.716527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.993774] alg: No test for adler32 (adler32-zlib) [ 2611.128706] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2611.334329] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2612.959386] Key type lgssc registered [ 2614.283835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.453682] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2622.465899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2639.007340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2645.156160] LustreError: 90666:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 2675.935283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.842223] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2685.434469] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2690.295281] LustreError: 91523:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2690.498679] Lustre: server umount lustre-OST0000 complete [ 2699.047416] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:48:07 (1758739687) [ 2704.861441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.151572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.677763] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2710.500036] LustreError: 92194:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.516961] LustreError: 92194:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2715.617865] LustreError: 92194:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.846629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2716.082158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2718.780827] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2723.021252] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2726.293572] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2732.662323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2733.033047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.266474] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2743.633232] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2750.452811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:577) [ 2760.681219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.687773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.689276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.718132] LustreError: 94600:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2763.724757] LustreError: 94600:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2763.892566] Lustre: server umount lustre-OST0000 complete [ 2767.328927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2767.333303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.340848] Lustre: Skipped 1 previous similar message [ 2767.344173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.346692] Lustre: Skipped 1 previous similar message [ 2770.913089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.917764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.283488] LustreError: 94802:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2773.287928] LustreError: 94802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2773.376753] Lustre: server umount lustre-MDT0000 complete [ 2776.037883] LustreError: 92190:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.051073] LustreError: 92190:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2776.852956] LustreError: 92174:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739766 with bad export cookie 15123018668835254423 [ 2776.860649] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2776.994743] LustreError: 95003:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2777.004227] LustreError: 95003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2777.289972] Lustre: server umount lustre-MDT0001 complete [ 2786.258420] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2789.094632] Key type lgssc unregistered [ 2789.451826] LNet: 95536:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.462696] LNetError: 95536:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2790.501967] LNet: Removed LNI 192.168.202.145@tcp [ 2791.283494] Key type .llcrypt unregistered [ 2791.285502] Key type ._llcrypt unregistered [ 2811.420243] Key type ._llcrypt registered [ 2811.422195] Key type .llcrypt registered [ 2811.503465] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2812.108963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2812.171051] alg: No test for adler32 (adler32-zlib) [ 2813.236344] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2813.452540] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2815.087405] Key type lgssc registered [ 2815.994357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2822.634848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2822.661035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.055183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.024973] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2833.380411] LustreError: 96933:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.396863] LustreError: 96933:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2834.387778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2834.616950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.184956] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2842.829862] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2846.944283] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2853.875164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2854.231319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.683217] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 2863.133187] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2868.594292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:609) [ 2878.945826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2878.949413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.957115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.974855] Lustre: Skipped 1 previous similar message [ 2878.987647] Lustre: Skipped 1 previous similar message [ 2884.075650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.083464] Lustre: Skipped 1 previous similar message [ 2889.189333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.200364] Lustre: Skipped 1 previous similar message [ 2890.207512] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2890.242292] LustreError: 99199:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2890.362753] Lustre: server umount lustre-OST0000 complete [ 2894.306496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.325604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.328464] Lustre: Skipped 1 previous similar message [ 2899.426402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.441156] Lustre: Skipped 1 previous similar message [ 2899.634430] LustreError: 99404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2899.636568] LustreError: 99404:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2899.705716] Lustre: server umount lustre-MDT0000 complete [ 2903.301396] LustreError: 96913:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758739893 with bad export cookie 9260075398964771435 [ 2903.313601] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2903.462844] LustreError: 99605:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2903.465934] LustreError: 99605:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2903.679327] Lustre: server umount lustre-MDT0001 complete [ 2914.537875] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 2916.681246] Key type lgssc unregistered [ 2916.923885] LNet: 100139:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2916.933651] LNetError: 100139:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2917.989710] LNet: Removed LNI 192.168.202.145@tcp [ 2918.714504] Key type .llcrypt unregistered [ 2918.716562] Key type ._llcrypt unregistered [ 2934.599355] Key type ._llcrypt registered [ 2934.600966] Key type .llcrypt registered [ 2934.707907] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 2935.542253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2935.560460] alg: No test for adler32 (adler32-zlib) [ 2936.541767] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 2936.723948] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 2938.367166] Key type lgssc registered [ 2939.284376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2945.683982] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2945.706803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2962.271347] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2968.415246] LustreError: 101362:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 2998.179470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.601669] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3006.647737] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3012.900603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.173439] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3013.180275] LustreError: 102275:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3013.187651] LustreError: 102275:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3013.194971] LustreError: 102275:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3013.200510] LustreError: 101370:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740003 with bad export cookie 15217926879032066831 [ 3013.210916] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3013.337993] Lustre: server umount lustre-MDT0000 complete [ 3013.346118] LustreError: 102275:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3020.199024] LustreError: 102705:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3020.364665] Lustre: server umount lustre-OST0000 complete [ 3027.094262] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 3033.911452] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3041.181362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3045.855087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3050.667791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3056.037533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3061.760751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3061.804598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.967981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3062.008476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3062.059789] Lustre: lustre-MDT0000: new disk, initializing [ 3062.127487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.144674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3065.022482] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3073.752657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3073.810953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3073.892753] Lustre: 105056:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3073.958900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3073.961555] Lustre: Skipped 1 previous similar message [ 3074.028219] Lustre: lustre-MDT0001: new disk, initializing [ 3074.130240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3074.158195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3074.163217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3077.369542] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3083.256902] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3086.484703] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3092.006569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3092.081326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3092.269618] Lustre: lustre-OST0000: new disk, initializing [ 3092.275620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3092.355766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.065753] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3101.250599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3101.272189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3101.373638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3103.624855] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3107.237387] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3107.415411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3110.791362] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3111.011553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3117.024909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.025710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.033530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.043135] Lustre: Skipped 1 previous similar message [ 3117.056219] Lustre: Skipped 1 previous similar message [ 3118.586063] LustreError: 107993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3118.589041] LustreError: 107993:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3118.770132] Lustre: server umount lustre-OST0000 complete [ 3122.151700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.160735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3122.169807] Lustre: Skipped 1 previous similar message [ 3127.266572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.278549] Lustre: Skipped 2 previous similar messages [ 3128.176857] LustreError: 108196:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3128.178966] LustreError: 108196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3128.287442] Lustre: server umount lustre-MDT0000 complete [ 3131.585352] LustreError: 105050:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740121 with bad export cookie 15217926879032066922 [ 3131.586722] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3131.598388] LustreError: 105050:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3131.766374] LustreError: 108398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3131.781572] LustreError: 108398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3132.043804] Lustre: server umount lustre-MDT0001 complete [ 3140.419640] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:55:28 (1758740128) [ 3143.407548] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3157.152873] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 3164.072351] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3172.019301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3178.558312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3185.332351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3192.032388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.286241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.388904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.614816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3199.648392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3199.722878] Lustre: lustre-MDT0000: new disk, initializing [ 3199.821772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.857937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3203.712341] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3213.681512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.776298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.877941] Lustre: 111534:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3213.933281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3213.937156] Lustre: Skipped 1 previous similar message [ 3214.005498] Lustre: lustre-MDT0001: new disk, initializing [ 3214.110945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3214.150373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3214.161847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3217.755843] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3224.609665] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3228.151837] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3234.529095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3234.635733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3234.845090] Lustre: lustre-OST0000: new disk, initializing [ 3234.850464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3234.922817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.461903] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3243.606624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3243.612121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3243.713501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3246.605713] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3249.573177] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3249.754947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3253.158173] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3253.339469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3259.365929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3259.373070] LustreError: Skipped 1 previous similar message [ 3259.378538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.388566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3259.393926] Lustre: Skipped 1 previous similar message [ 3264.487953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.491595] Lustre: Skipped 1 previous similar message [ 3269.087123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3269.131067] LustreError: 114474:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3269.136047] LustreError: 114474:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3269.272100] Lustre: server umount lustre-OST0000 complete [ 3274.722656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.726960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.741596] Lustre: Skipped 2 previous similar messages [ 3278.171216] LustreError: 114677:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3278.173919] LustreError: 114677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3278.256527] Lustre: server umount lustre-MDT0000 complete [ 3281.257075] LustreError: 113281:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740271 with bad export cookie 15217926879032067804 [ 3281.262585] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.264886] LustreError: 113281:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3281.588736] Lustre: server umount lustre-MDT0001 complete [ 3288.571505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.868392] LustreError: 115355:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.873193] LustreError: 115355:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3288.943181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.208299] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3294.177702] LustreError: 115355:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.274952] LustreError: 115354:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.845942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3302.664228] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3307.074490] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3310.074810] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3315.368482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3320.004664] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3324.592036] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3339.190697] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3344.358543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.361692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3344.366072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3344.366083] Lustre: Skipped 1 previous similar message [ 3344.371199] Lustre: Skipped 1 previous similar message [ 3344.380679] LustreError: Skipped 1 previous similar message [ 3348.038144] LustreError: 117718:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3348.049185] LustreError: 117718:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3348.198973] Lustre: server umount lustre-OST0000 complete [ 3354.592820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.602282] Lustre: Skipped 1 previous similar message [ 3357.465904] Lustre: server umount lustre-MDT0000 complete [ 3359.715675] LustreError: 116066:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.245521] LustreError: 115335:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740350 with bad export cookie 15217926879032068679 [ 3360.252466] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3360.565139] Lustre: server umount lustre-MDT0001 complete [ 3368.148229] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 3370.115818] Key type lgssc unregistered [ 3370.361741] LNet: 118654:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3370.372094] LNetError: 118654:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3370.385899] LNet: Removed LNI 192.168.202.145@tcp [ 3370.983086] Key type .llcrypt unregistered [ 3370.984821] Key type ._llcrypt unregistered [ 3383.858945] Key type ._llcrypt registered [ 3383.861763] Key type .llcrypt registered [ 3383.934172] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 3394.793224] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 3395.650260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3395.663331] alg: No test for adler32 (adler32-zlib) [ 3396.783500] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 3396.953587] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 3398.647177] Key type lgssc registered [ 3399.677536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3406.397153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3411.505908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3416.172256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3421.265961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3426.663827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3426.729834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.743840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.930486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3427.956950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3428.011612] Lustre: lustre-MDT0000: new disk, initializing [ 3428.082777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.099927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3431.364683] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3440.442989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3440.525416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3440.620185] Lustre: 121761:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3440.683458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3440.685660] Lustre: Skipped 1 previous similar message [ 3440.731749] Lustre: lustre-MDT0001: new disk, initializing [ 3440.822938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3440.866746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3440.878758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3444.083623] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3450.255369] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3453.434227] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3458.580530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3458.634938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3458.845234] Lustre: lustre-OST0000: new disk, initializing [ 3458.855921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3458.961515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.920763] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3465.313776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3465.325662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3465.387432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3470.145767] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3473.917580] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3474.115180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3477.331672] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3477.489564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3481.059189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.062189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.062554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.067577] Lustre: Skipped 1 previous similar message [ 3486.178531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.181418] Lustre: Skipped 2 previous similar messages [ 3491.299180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.345231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3493.395779] LustreError: 124709:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3493.554868] Lustre: server umount lustre-OST0000 complete [ 3497.440757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3497.452687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.469624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3497.472585] Lustre: Skipped 1 previous similar message [ 3501.538801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.539752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.758906] LustreError: 124912:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3502.766426] LustreError: 124912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3502.937058] Lustre: server umount lustre-MDT0000 complete [ 3506.224349] LustreError: 121753:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740496 with bad export cookie 5249208279042198413 [ 3506.227523] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3506.265711] LustreError: 121753:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3506.383857] LustreError: 125115:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3506.387850] LustreError: 125115:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3506.568341] Lustre: server umount lustre-MDT0001 complete [ 3512.711136] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:01:41 (1758740501) [ 3517.434484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.868255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.719416] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3523.041905] LustreError: 125789:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.053018] LustreError: 125789:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3526.879801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3527.160223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.032225] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3533.763714] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3536.442848] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3542.497926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3542.498173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.502647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3542.502658] Lustre: Skipped 1 previous similar message [ 3542.527080] Lustre: Skipped 1 previous similar message [ 3543.827091] LustreError: 127131:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3543.832541] LustreError: 127131:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3543.894608] Lustre: server umount lustre-MDT0000 complete [ 3546.126581] LustreError: 125768:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740536 with bad export cookie 5249208279042199295 [ 3546.131904] LustreError: 125768:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3546.132662] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3546.342113] Lustre: server umount lustre-MDT0001 complete [ 3552.122209] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:02:20 (1758740540) [ 3557.258470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3574.815655] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3580.959489] LustreError: 127978:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 3609.695300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.218470] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3618.788716] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3620.339945] LustreError: 128642:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3620.349386] LustreError: 128642:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3620.493571] Lustre: server umount lustre-OST0000 complete [ 3626.914184] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:03:35 (1758740615) [ 3632.193248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.512913] LustreError: 129313:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.579628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.792530] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3637.734413] LustreError: 129313:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.103098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3645.735639] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3649.877112] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3652.943213] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3657.900649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3662.070912] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3665.692867] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3677.667158] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.669690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.682302] Lustre: Skipped 2 previous similar messages [ 3677.683547] Lustre: Skipped 2 previous similar messages [ 3681.945531] LustreError: 131557:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3681.948646] LustreError: 131557:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3682.015495] Lustre: server umount lustre-MDT0000 complete [ 3682.788136] LustreError: 129309:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.795203] LustreError: 129309:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3684.405541] LustreError: 129293:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740674 with bad export cookie 5249208279042199960 [ 3684.409077] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3684.412492] LustreError: 129293:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3684.674546] Lustre: server umount lustre-MDT0001 complete [ 3697.176690] Lustre: server umount lustre-OST0000 complete [ 3703.304133] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:04:52 (1758740692) [ 3708.102667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.483149] LustreError: 132633:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.538905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.542897] Lustre: Skipped 2 previous similar messages [ 3711.222085] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3717.115493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3720.305160] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3723.733405] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3725.996714] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3730.535105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3733.867788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3734.560670] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3738.044787] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3740.615796] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3740.772105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3743.394296] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3743.525314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3747.300703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3747.300978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.305180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3747.309624] Lustre: Skipped 1 previous similar message [ 3747.311868] Lustre: Skipped 1 previous similar message [ 3750.898814] LustreError: 135375:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3750.901867] LustreError: 135375:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3751.005586] Lustre: server umount lustre-OST0000 complete [ 3759.731198] Lustre: server umount lustre-MDT0000 complete [ 3762.550291] LustreError: 132613:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740752 with bad export cookie 5249208279042200982 [ 3762.552356] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3762.570042] LustreError: 132613:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3762.921661] Lustre: server umount lustre-MDT0001 complete [ 3768.978845] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:05:57 (1758740757) [ 3773.462328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3791.007649] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3797.151489] LustreError: 136423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 3826.911361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.913573] Lustre: Skipped 2 previous similar messages [ 3829.954842] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3833.088322] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3837.440717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.702236] LustreError: 136450:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.714939] LustreError: 136450:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3837.827499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3840.328917] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3845.497839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3848.187911] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3851.881994] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3853.880137] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3856.522235] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3856.669541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3859.361619] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3859.499127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3860.964464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3860.967208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.968672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.968679] Lustre: Skipped 3 previous similar messages [ 3861.019585] Lustre: Skipped 4 previous similar messages [ 3866.698149] LustreError: 139208:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3866.703195] LustreError: 139208:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3866.745211] Lustre: server umount lustre-MDT0000 complete [ 3868.907425] LustreError: 136432:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740858 with bad export cookie 5249208279042202011 [ 3868.915089] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.114434] Lustre: server umount lustre-MDT0001 complete [ 3881.439671] Lustre: server umount lustre-OST0000 complete [ 3885.888526] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:07:55 (1758740875) [ 3889.996793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3907.551592] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3913.695510] LustreError: 140257:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 3945.935233] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3949.217465] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3952.793723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3953.050265] LustreError: 140283:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.060888] LustreError: 140283:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3953.133694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3955.348424] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3959.947068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3960.152651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3960.155844] Lustre: Skipped 4 previous similar messages [ 3961.999512] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3964.818563] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3966.894274] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3970.764389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3970.798909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3970.880205] Lustre: lustre-OST0001: new disk, initializing [ 3970.883224] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3974.191369] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 3975.709665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3975.715857] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3975.804130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3978.965244] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3981.259578] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3981.426366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3983.312498] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3983.431683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3985.897927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3985.898378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.910755] Lustre: Skipped 1 previous similar message [ 3990.509149] LustreError: 143988:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3990.512463] LustreError: 143988:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3990.533648] Lustre: server umount lustre-OST0000 complete [ 3991.013573] LustreError: 140274:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.020932] LustreError: 140274:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3996.137942] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3996.141559] Lustre: Skipped 8 previous similar messages [ 3998.470540] Lustre: server umount lustre-OST0001 complete [ 4006.783688] Lustre: server umount lustre-MDT0000 complete [ 4008.393393] LustreError: 141210:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758740998 with bad export cookie 5249208279042202956 [ 4008.397039] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4008.398328] LustreError: 141210:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4008.568683] Lustre: server umount lustre-MDT0001 complete [ 4011.226763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4013.054628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4014.839811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4016.841953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4020.284884] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:10:09 (1758741009) [ 4021.032487] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4021.895549] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4022.747601] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:10:12 (1758741012) [ 4025.959881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.132351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4026.152455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4028.214953] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4032.957757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4033.009272] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4033.027760] Lustre: 146227:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4035.255504] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4037.857867] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4039.579228] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4043.189315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4043.323370] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4046.108776] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4048.838351] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4050.922925] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4053.537178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4054.150940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4056.036464] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4056.135470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4058.594864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4058.595520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.597972] LustreError: Skipped 3 previous similar messages [ 4058.603277] Lustre: Skipped 4 previous similar messages [ 4063.172617] Lustre: server umount lustre-OST0000 complete [ 4079.075669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4081.685790] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4084.028976] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4086.024105] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4098.663104] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.671638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.673828] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4098.687335] Lustre: Skipped 1 previous similar message [ 4099.639523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 13 sec [ 4101.665122] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4101.769937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4103.506606] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af890083000.ost_server_uuid 50 [ 4104.245630] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9af890083000.ost_server_uuid in FULL state after 0 sec [ 4104.327058] LustreError: 146234:0:(ldlm_lockd.c:728:ldlm_handle_ast_error()) ### client (nid 192.168.202.45@tcp) returned error from blocking AST (req@0000000038ef24fb x1844172959203200 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b222dda7/0x48d8ef1e4859a683 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.45@tcp remote: 0x84a8842951b1e221 expref: 5 pid: 146234 timeout: 4204 lvb_type: 0 [ 4104.341052] LustreError: lustre-MDT0000: A client on nid 192.168.202.45@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4104.345435] LustreError: 146222:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.45@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b222dda7/0x48d8ef1e4859a683 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.45@tcp remote: 0x84a8842951b1e221 expref: 6 pid: 146234 timeout: 0 lvb_type: 0 [ 4109.283190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.289320] LustreError: Skipped 2 previous similar messages [ 4112.077356] Lustre: server umount lustre-OST0000 complete [ 4119.814206] LustreError: 151090:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4119.817398] LustreError: 151090:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 4119.851644] Lustre: server umount lustre-MDT0000 complete [ 4121.596376] LustreError: 146219:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741111 with bad export cookie 5249208279042204132 [ 4121.597851] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4121.602418] LustreError: 146219:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4127.866211] Lustre: server umount lustre-MDT0001 complete [ 4132.375488] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4133.709679] Key type lgssc unregistered [ 4133.870756] LNet: 151823:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4133.873869] LNetError: 151823:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4133.882402] LNet: Removed LNI 192.168.202.145@tcp [ 4134.260582] Key type .llcrypt unregistered [ 4134.262312] Key type ._llcrypt unregistered [ 4145.842073] Key type ._llcrypt registered [ 4145.844222] Key type .llcrypt registered [ 4145.905243] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4146.351405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4146.415831] alg: No test for adler32 (adler32-zlib) [ 4147.368940] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4147.520193] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4149.119187] Key type lgssc registered [ 4149.716703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4153.670170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4153.683046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.905129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.553783] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4160.801139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4160.994808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4162.751119] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4165.410742] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4167.155109] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4170.427084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4170.605161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.511682] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4176.386188] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4179.826458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4190.099572] Lustre: Failing over lustre-MDT0000 [ 4190.223745] LustreError: 155463:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4190.283143] Lustre: server umount lustre-MDT0000 complete [ 4191.201351] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.205904] LustreError: 153191:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.209022] Lustre: Skipped 1 previous similar message [ 4191.221716] LustreError: 153191:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4196.323720] LustreError: 153913:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.334268] LustreError: 153913:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 4201.441731] LustreError: 153191:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.453032] LustreError: 153191:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4201.597137] LustreError: 153174:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.45@tcp arrived at 1758741191 with bad export cookie 15760902767881109760 [ 4201.608154] LustreError: 153174:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4204.928411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.996344] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.127315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.158194] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4206.879850] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4207.894957] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4207.898862] Lustre: lustre-MDT0000: Denying connection for new client 843c02c7-e210-4680-b582-f971ee9eeb8b (at 192.168.202.45@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4210.148583] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4215.266257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.268304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.272083] Lustre: Skipped 2 previous similar messages [ 4220.386798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4220.390413] Lustre: Skipped 2 previous similar messages [ 4220.974118] LustreError: 156394:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4220.977301] LustreError: 156394:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4221.054714] Lustre: server umount lustre-OST0000 complete [ 4222.588556] LustreError: 156595:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4222.592463] Lustre: 155961:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4222.595857] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4222.604497] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4222.609891] LustreError: 155961:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b78bbe73 x1844173746507392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4222.618801] LustreError: 155961:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4222.623844] LustreError: 155961:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4222.628643] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 4222.711193] LustreError: 156595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4222.713285] LustreError: 156595:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4222.745578] Lustre: server umount lustre-MDT0000 complete [ 4224.376104] LustreError: 154814:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741214 with bad export cookie 15760902767881110096 [ 4224.380961] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.381555] LustreError: 154814:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4224.382243] LustreError: 152504:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4230.536860] LustreError: 156796:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4230.540234] LustreError: 156796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4230.643568] Lustre: server umount lustre-MDT0001 complete [ 4235.553493] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4236.950385] Key type lgssc unregistered [ 4237.119076] LNet: 157328:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4237.123484] LNetError: 157328:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4237.130735] LNet: Removed LNI 192.168.202.145@tcp [ 4237.518548] Key type .llcrypt unregistered [ 4237.520152] Key type ._llcrypt unregistered [ 4248.268225] Key type ._llcrypt registered [ 4248.270023] Key type .llcrypt registered [ 4248.318700] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4248.677472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4248.728741] alg: No test for adler32 (adler32-zlib) [ 4249.602864] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4249.697449] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4251.287210] Key type lgssc registered [ 4251.788692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4255.158332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4255.168189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.382803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4257.908578] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4261.296103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4261.471120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4263.171572] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4265.555588] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4267.204728] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4270.101984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4270.263290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.917653] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4275.375336] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4282.534311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4287.969281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.970132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4287.976125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4287.981759] Lustre: Skipped 1 previous similar message [ 4289.714127] LustreError: 160965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4289.798888] Lustre: server umount lustre-OST0000 complete [ 4292.064397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4292.067675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.072300] Lustre: Skipped 1 previous similar message [ 4292.074636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.088148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.088922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.568117] LustreError: 161166:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4293.571161] LustreError: 161166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4293.618235] Lustre: server umount lustre-MDT0000 complete [ 4295.189382] LustreError: 158679:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741285 with bad export cookie 17989846323721125064 [ 4295.192108] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.194656] LustreError: 158679:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4295.264896] LustreError: 161366:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4295.267908] LustreError: 161366:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4295.361303] Lustre: server umount lustre-MDT0001 complete [ 4299.557722] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4300.811774] Key type lgssc unregistered [ 4300.980706] LNet: 161898:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4300.984957] LNetError: 161898:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4300.990693] LNet: Removed LNI 192.168.202.145@tcp [ 4301.366428] Key type .llcrypt unregistered [ 4301.367919] Key type ._llcrypt unregistered [ 4316.072454] Key type ._llcrypt registered [ 4316.074122] Key type .llcrypt registered [ 4316.121867] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4316.587812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.600268] alg: No test for adler32 (adler32-zlib) [ 4317.487743] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4317.587951] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4319.183540] Key type lgssc registered [ 4319.650463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4322.978453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4322.989031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4324.206837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.812539] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4329.494675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4329.657604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.289500] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4333.564895] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4335.051377] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4337.792903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4337.942745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.442202] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4342.736129] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4349.224684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4355.552128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4355.554921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.559670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.649799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4359.650719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.657378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.749139] LustreError: 165824:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4360.818867] Lustre: server umount lustre-OST0000 complete [ 4364.769367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4364.770167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4364.774978] Lustre: Skipped 1 previous similar message [ 4364.777693] Lustre: Skipped 1 previous similar message [ 4368.384497] LustreError: 166026:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4368.387607] LustreError: 166026:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4368.414140] Lustre: server umount lustre-MDT0000 complete [ 4369.888035] LustreError: 163557:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.896865] LustreError: 163557:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4369.973759] LustreError: 163539:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741359 with bad export cookie 2502644454965973944 [ 4369.977308] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4369.979094] LustreError: 163539:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4370.048296] LustreError: 166228:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4370.051257] LustreError: 166228:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4370.144951] Lustre: server umount lustre-MDT0001 complete [ 4374.264483] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4375.437126] Key type lgssc unregistered [ 4375.589523] LNet: 166759:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4375.593320] LNetError: 166759:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4375.601331] LNet: Removed LNI 192.168.202.145@tcp [ 4375.963793] Key type .llcrypt unregistered [ 4375.965334] Key type ._llcrypt unregistered [ 4385.552917] Key type ._llcrypt registered [ 4385.554646] Key type .llcrypt registered [ 4385.600217] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4385.990543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4386.025689] alg: No test for adler32 (adler32-zlib) [ 4386.906915] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4387.004472] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4388.599255] Key type lgssc registered [ 4389.055360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4393.726529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4393.736529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.925142] Lustre: *** cfs_fail_loc=135, val=0*** [ 4394.969681] LustreError: 168240:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4394.972623] LustreError: 168240:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.145@tcp: cfg command failed: rc = -2 [ 4394.975398] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4394.975398] [ 4394.978613] LustreError: MGC192.168.202.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4394.983835] LustreError: 168200:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4394.986906] LustreError: 168200:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4394.989447] LustreError: 168200:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4394.994105] LustreError: 168200:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4395.067758] Lustre: server umount lustre-MDT0000 complete [ 4395.069780] LustreError: 168200:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4397.922551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.130041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.725189] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4403.175582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4403.351701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.015452] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4407.343424] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4408.937727] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4413.920900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4413.921094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.930318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.933059] Lustre: Skipped 1 previous similar message [ 4419.042138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.046543] Lustre: Skipped 1 previous similar message [ 4424.159286] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4424.161428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4424.167815] Lustre: Skipped 1 previous similar message [ 4424.204763] LustreError: 169954:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4424.208825] LustreError: 169954:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4424.240274] Lustre: server umount lustre-MDT0000 complete [ 4425.919041] LustreError: 168593:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741415 with bad export cookie 2561675240845288657 [ 4425.921226] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.926121] LustreError: 168593:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4426.009928] LustreError: 170157:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4426.013068] LustreError: 170157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4426.105596] Lustre: server umount lustre-MDT0001 complete [ 4430.441768] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4431.626898] Key type lgssc unregistered [ 4431.777409] LNet: 170688:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4431.780944] LNetError: 170688:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4431.789408] LNet: Removed LNI 192.168.202.145@tcp [ 4432.116779] Key type .llcrypt unregistered [ 4432.118316] Key type ._llcrypt unregistered [ 4440.436625] Key type ._llcrypt registered [ 4440.438097] Key type .llcrypt registered [ 4440.486434] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4440.616587] Key type .llcrypt unregistered [ 4440.618422] Key type ._llcrypt unregistered [ 4448.203373] Key type ._llcrypt registered [ 4448.205131] Key type .llcrypt registered [ 4448.248750] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4448.709262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4448.719754] alg: No test for adler32 (adler32-zlib) [ 4449.590911] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4449.682678] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4451.271190] Key type lgssc registered [ 4451.707223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4454.722633] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4454.732832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4471.199493] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4477.343157] LustreError: 172755:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.145@tcp: failed processing log, type 4: rc = -110 [ 4507.039536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.369683] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4511.676318] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4514.561859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.834108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4514.864833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4516.645166] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4519.907092] LustreError: 172781:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.913490] LustreError: 172781:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4520.468225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4520.665813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4522.452560] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4524.991991] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4526.621482] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4527.867212] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4533.219778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.227415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.230367] Lustre: Skipped 1 previous similar message [ 4537.965471] LustreError: 175336:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4538.041526] Lustre: server umount lustre-OST0000 complete [ 4541.408230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4541.410995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.415912] Lustre: Skipped 1 previous similar message [ 4541.418519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.456649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.456675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.795031] LustreError: 175537:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4545.797390] LustreError: 175537:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4545.839289] Lustre: server umount lustre-MDT0000 complete [ 4547.442952] LustreError: 172764:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741537 with bad export cookie 13744836944028685041 [ 4547.448170] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4547.448307] LustreError: 172764:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4547.517534] LustreError: 175738:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4547.520509] LustreError: 175738:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4547.622930] Lustre: server umount lustre-MDT0001 complete [ 4552.059594] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4553.313212] Key type lgssc unregistered [ 4553.463638] LNet: 176269:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4553.467905] LNetError: 176269:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4553.475577] LNet: Removed LNI 192.168.202.145@tcp [ 4553.824448] Key type .llcrypt unregistered [ 4553.826056] Key type ._llcrypt unregistered [ 4563.826289] Key type ._llcrypt registered [ 4563.827532] Key type .llcrypt registered [ 4563.884328] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4564.259724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4564.321132] alg: No test for adler32 (adler32-zlib) [ 4565.202971] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4565.301739] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4566.895204] Key type lgssc registered [ 4567.335161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4570.435235] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4570.448079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.653752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4573.107414] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4576.537915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4576.690058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4578.231724] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4580.289872] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4581.697495] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4584.210630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4584.369611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.481151] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4588.560448] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4596.647584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4600.369736] Lustre: Failing over lustre-MDT0000 [ 4600.441675] LustreError: 179958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4600.476858] Lustre: server umount lustre-MDT0000 complete [ 4601.826599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.827627] LustreError: 178895:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.832489] Lustre: Skipped 2 previous similar messages [ 4601.844389] LustreError: 178895:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 4604.026411] LustreError: 177664:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.947498] LustreError: 177636:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.955183] LustreError: 177636:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4609.136541] LustreError: 179258:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.091751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.153585] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.265405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.292103] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4615.882943] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4617.747990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4619.376352] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4619.749074] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4619.756599] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4619.772955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4623.863626] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4627.424334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4627.427641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.433552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4629.986287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4629.987132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4629.988039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.989131] Lustre: Skipped 1 previous similar message [ 4629.991654] LustreError: Skipped 1 previous similar message [ 4632.367146] LustreError: 181282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4632.370427] LustreError: 181282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4632.435878] Lustre: server umount lustre-OST0000 complete [ 4635.105040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4635.105215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.105776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4635.114884] Lustre: Skipped 1 previous similar message [ 4640.005119] LustreError: 181487:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4640.007714] LustreError: 181487:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4640.044916] Lustre: server umount lustre-MDT0000 complete [ 4640.225061] LustreError: 178895:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4640.231945] LustreError: 178895:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 4641.595554] LustreError: 177620:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741631 with bad export cookie 777623101882283152 [ 4641.598796] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4641.600659] LustreError: 177620:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4641.755587] Lustre: server umount lustre-MDT0001 complete [ 4645.950473] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4647.147832] Key type lgssc unregistered [ 4647.304440] LNet: 182219:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4647.307847] LNetError: 182219:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4647.318361] LNet: Removed LNI 192.168.202.145@tcp [ 4647.645604] Key type .llcrypt unregistered [ 4647.646905] Key type ._llcrypt unregistered [ 4657.151447] Key type ._llcrypt registered [ 4657.152836] Key type .llcrypt registered [ 4657.193134] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4657.615372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4657.630019] alg: No test for adler32 (adler32-zlib) [ 4658.502068] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4658.604113] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4660.183116] Key type lgssc registered [ 4660.575486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4663.368638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4663.377354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.559664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.956633] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4668.947105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4669.067886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4670.438593] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4672.370666] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4673.596030] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4675.892286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4676.023192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.988532] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4679.820564] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4689.322152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4691.225709] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4698.831314] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4708.965155] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4720.098296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.098987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4720.104972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4720.107281] LustreError: Skipped 1 previous similar message [ 4720.111708] Lustre: Skipped 1 previous similar message [ 4723.180163] LustreError: 186033:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4723.242086] Lustre: server umount lustre-OST0000 complete [ 4725.217442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.218091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4725.225039] Lustre: Skipped 2 previous similar messages [ 4730.336118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4730.338606] Lustre: Skipped 2 previous similar messages [ 4730.751317] LustreError: 186234:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4730.754019] LustreError: 186234:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4730.788090] Lustre: server umount lustre-MDT0000 complete [ 4732.197521] LustreError: 183572:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741722 with bad export cookie 11854801622072873620 [ 4732.201401] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4732.201955] LustreError: 183572:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4732.273564] LustreError: 186436:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4732.276281] LustreError: 186436:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4732.364489] Lustre: server umount lustre-MDT0001 complete [ 4736.423890] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4737.633422] Key type lgssc unregistered [ 4737.775658] LNet: 186967:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.778972] LNetError: 186967:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.790620] LNet: Removed LNI 192.168.202.145@tcp [ 4738.151412] Key type .llcrypt unregistered [ 4738.152513] Key type ._llcrypt unregistered [ 4747.055563] Key type ._llcrypt registered [ 4747.056848] Key type .llcrypt registered [ 4747.093840] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4747.507528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4747.514884] alg: No test for adler32 (adler32-zlib) [ 4748.362283] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4748.447408] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4750.031164] Key type lgssc registered [ 4750.424937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4753.275740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4753.283701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4754.448093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.804414] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4758.719733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4758.829992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4760.121103] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4762.012443] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4763.271794] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4765.520612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4765.652985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4767.696529] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4769.634445] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4781.029051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4786.722303] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4794.541684] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4803.929768] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4814.142409] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4825.055964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4825.059832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.066073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4827.106290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4827.106701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4827.110266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4827.112108] Lustre: Skipped 1 previous similar message [ 4829.229204] LustreError: 192673:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4829.287996] Lustre: server umount lustre-OST0000 complete [ 4830.687887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4830.691342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.695881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4836.732287] LustreError: 192874:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4836.735278] LustreError: 192874:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4836.761868] Lustre: server umount lustre-MDT0000 complete [ 4837.344042] LustreError: 189048:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.350233] LustreError: 189048:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4838.055241] LustreError: 189055:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741828 with bad export cookie 1406117630981828437 [ 4838.057571] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4838.060642] LustreError: 189055:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4838.127031] LustreError: 193075:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4838.130037] LustreError: 193075:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4838.210570] Lustre: server umount lustre-MDT0001 complete [ 4841.828397] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4842.941095] Key type lgssc unregistered [ 4843.078395] LNet: 193606:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4843.080702] LNetError: 193606:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4843.090358] LNet: Removed LNI 192.168.202.145@tcp [ 4843.398878] Key type .llcrypt unregistered [ 4843.400282] Key type ._llcrypt unregistered [ 4852.928897] Key type ._llcrypt registered [ 4852.930033] Key type .llcrypt registered [ 4852.972180] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4853.423381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4853.430043] alg: No test for adler32 (adler32-zlib) [ 4854.304447] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4854.394240] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4855.991253] Key type lgssc registered [ 4856.371541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4859.253445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4859.261758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.446137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.903675] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4865.036838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4865.160979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4866.499259] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4868.487210] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4869.884293] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.357962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4872.494164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.510877] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4876.325517] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4885.798303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4891.106134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.107475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.108298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.109944] Lustre: Skipped 1 previous similar message [ 4891.113196] LustreError: Skipped 1 previous similar message [ 4896.107342] LustreError: 197393:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4896.168449] Lustre: server umount lustre-OST0000 complete [ 4901.344857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4901.344976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.345655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.345659] Lustre: Skipped 1 previous similar message [ 4901.354721] Lustre: Skipped 1 previous similar message [ 4903.618274] LustreError: 197595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4903.621090] LustreError: 197595:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4903.663170] Lustre: server umount lustre-MDT0000 complete [ 4905.018864] LustreError: 194956:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741894 with bad export cookie 15159796932854188183 [ 4905.023304] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.024079] LustreError: 194956:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4905.100234] LustreError: 197797:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4905.102299] LustreError: 197797:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4905.178125] Lustre: server umount lustre-MDT0001 complete [ 4908.893827] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4910.063235] Key type lgssc unregistered [ 4910.212623] LNet: 198328:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4910.216517] LNetError: 198328:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4910.230561] LNet: Removed LNI 192.168.202.145@tcp [ 4910.572680] Key type .llcrypt unregistered [ 4910.574182] Key type ._llcrypt unregistered [ 4918.086611] Key type ._llcrypt registered [ 4918.087585] Key type .llcrypt registered [ 4918.124393] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4918.456811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4918.502175] alg: No test for adler32 (adler32-zlib) [ 4919.383553] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4919.481466] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4921.071139] Key type lgssc registered [ 4921.465954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4924.383878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4924.394210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.598792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.017547] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4930.484918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4930.666313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4932.322755] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4934.417952] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4935.782729] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4938.460799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4938.605631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4940.688782] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4942.820954] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4950.886749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4961.248905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4961.250181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.252017] LustreError: Skipped 1 previous similar message [ 4961.257853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.260793] Lustre: Skipped 1 previous similar message [ 4963.181025] LustreError: 201975:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4963.248431] Lustre: server umount lustre-OST0000 complete [ 4966.368969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.369247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4966.370120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4966.374785] Lustre: Skipped 2 previous similar messages [ 4970.685204] LustreError: 202176:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4970.687505] LustreError: 202176:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4970.719528] Lustre: server umount lustre-MDT0000 complete [ 4971.488129] LustreError: 199553:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4971.493578] LustreError: 199553:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4972.192621] LustreError: 199535:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758741962 with bad export cookie 4979334179254073695 [ 4972.196370] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4972.198333] LustreError: 199535:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4972.275447] LustreError: 202377:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4972.277347] LustreError: 202377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4972.380400] Lustre: server umount lustre-MDT0001 complete [ 4976.627931] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 4977.869352] Key type lgssc unregistered [ 4978.033744] LNet: 202907:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4978.037649] LNetError: 202907:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4978.050393] LNet: Removed LNI 192.168.202.145@tcp [ 4978.419768] Key type .llcrypt unregistered [ 4978.421688] Key type ._llcrypt unregistered [ 4986.274559] Key type ._llcrypt registered [ 4986.276042] Key type .llcrypt registered [ 4986.318105] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 4986.634499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4986.654107] alg: No test for adler32 (adler32-zlib) [ 4987.521989] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 4987.612839] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 4989.207199] Key type lgssc registered [ 4989.626675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4992.511286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4992.518819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4993.686723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.090558] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 4998.211902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4998.355438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4999.799710] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5001.717554] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5003.160457] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5005.586942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5005.719457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5007.740951] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5009.697121] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5019.046106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5022.913178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5022.942818] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5023.042037] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.035719] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5026.922675] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5027.506372] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5037.896633] Lustre: Permanently deactivating lustre-OST0001 [ 5037.902938] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5038.779803] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5048.807486] Lustre: Permanently reactivating lustre-OST0001 [ 5048.814655] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5048.817490] Lustre: Skipped 2 previous similar messages [ 5057.528659] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.533605] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5057.536066] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5057.540703] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5057.553626] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5062.624196] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5062.624255] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.627256] LustreError: Skipped 1 previous similar message [ 5062.631249] Lustre: Skipped 1 previous similar message [ 5062.633756] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5062.637264] Lustre: Skipped 1 previous similar message [ 5065.186584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5065.189319] Lustre: Skipped 1 previous similar message [ 5066.412056] LustreError: 207602:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5066.427873] Lustre: server umount lustre-OST0001 complete [ 5070.306045] LustreError: 205728:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.307597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5070.309941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.311593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5070.313255] LustreError: 205728:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5070.316215] LustreError: Skipped 1 previous similar message [ 5073.712494] LustreError: 207803:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5073.715085] LustreError: 207803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5073.779151] Lustre: server umount lustre-OST0000 complete [ 5075.424355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5075.424670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5075.430526] Lustre: Skipped 1 previous similar message [ 5075.432519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5075.434938] Lustre: Skipped 2 previous similar messages [ 5080.544378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5080.547079] Lustre: Skipped 1 previous similar message [ 5081.221593] LustreError: 208004:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5081.224467] LustreError: 208004:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5081.271655] Lustre: server umount lustre-MDT0000 complete [ 5082.672232] LustreError: 205756:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742072 with bad export cookie 13238469885947475182 [ 5082.675708] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5082.676812] LustreError: 205756:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5082.841336] Lustre: server umount lustre-MDT0001 complete [ 5086.849240] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5087.997430] Key type lgssc unregistered [ 5088.141570] LNet: 208736:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5088.144282] LNetError: 208736:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5088.151428] LNet: Removed LNI 192.168.202.145@tcp [ 5088.501689] Key type .llcrypt unregistered [ 5088.503170] Key type ._llcrypt unregistered [ 5101.124715] Key type ._llcrypt registered [ 5101.126159] Key type .llcrypt registered [ 5101.169271] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5107.243876] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5107.582736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5107.628117] alg: No test for adler32 (adler32-zlib) [ 5108.485180] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5108.581816] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5110.167163] Key type lgssc registered [ 5110.547561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5113.843497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5116.282681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5118.421821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5120.676870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.102119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5123.123467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5123.130410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.216877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5124.230662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5124.270781] Lustre: lustre-MDT0000: new disk, initializing [ 5124.297325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5124.305304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5125.737621] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5130.245343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.272029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5130.308138] Lustre: 212653:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.337756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5130.340311] Lustre: Skipped 1 previous similar message [ 5130.378871] Lustre: lustre-MDT0001: new disk, initializing [ 5130.419035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.432351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5130.435509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.837641] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5135.022642] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5136.282264] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5138.607323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5138.635196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5138.732740] Lustre: lustre-OST0000: new disk, initializing [ 5138.734655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5138.766939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5140.721717] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5143.850851] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5145.324077] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5147.129133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5147.133097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5147.173129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5147.461845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5148.849597] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5148.931140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5152.227349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5152.227690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.230324] LustreError: Skipped 1 previous similar message [ 5152.230486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5152.234121] Lustre: Skipped 1 previous similar message [ 5152.239116] Lustre: Skipped 1 previous similar message [ 5155.692652] LustreError: 215608:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5155.765525] Lustre: server umount lustre-OST0000 complete [ 5157.344345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.345055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.348268] Lustre: Skipped 1 previous similar message [ 5162.464046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5162.466320] Lustre: Skipped 2 previous similar messages [ 5163.198680] LustreError: 215810:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5163.201209] LustreError: 215810:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5163.235802] Lustre: server umount lustre-MDT0000 complete [ 5164.470223] LustreError: 213436:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742154 with bad export cookie 12399723971440282562 [ 5164.472805] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5164.474539] LustreError: 213436:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5164.530991] LustreError: 216011:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5164.532962] LustreError: 216011:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5164.607630] Lustre: server umount lustre-MDT0001 complete [ 5167.373957] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 15:29:16 (1758742156) [ 5169.799848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.998331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.332845] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5174.097840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5174.210386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5175.540576] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5177.388264] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5178.616605] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5180.881031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5182.892250] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5184.688468] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5195.953294] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5200.410560] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5207.898686] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5217.461514] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5234.573488] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5234.575544] Lustre: Skipped 1 previous similar message [ 5257.102983] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5257.105956] Lustre: Skipped 2 previous similar messages [ 5293.719613] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5293.721714] Lustre: Skipped 4 previous similar messages [ 5358.561076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.563105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.564197] LustreError: Skipped 1 previous similar message [ 5358.564202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.568231] Lustre: Skipped 1 previous similar message [ 5363.682037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.684124] Lustre: Skipped 2 previous similar messages [ 5364.016052] LustreError: 220153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5364.019163] LustreError: 220153:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5364.088683] Lustre: server umount lustre-OST0000 complete [ 5368.800763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5368.800899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.807399] Lustre: Skipped 1 previous similar message [ 5371.518347] LustreError: 220355:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5371.520063] LustreError: 220355:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5371.549621] Lustre: server umount lustre-MDT0000 complete [ 5372.763987] LustreError: 216707:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742362 with bad export cookie 12399723971440283430 [ 5372.767217] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5372.767708] LustreError: 216707:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5372.896263] Lustre: server umount lustre-MDT0001 complete [ 5376.360915] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5377.420279] Key type lgssc unregistered [ 5377.550299] LNet: 221088:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5377.552652] LNetError: 221088:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5377.563272] LNet: Removed LNI 192.168.202.145@tcp [ 5377.893443] Key type .llcrypt unregistered [ 5377.894809] Key type ._llcrypt unregistered [ 5386.882892] Key type ._llcrypt registered [ 5386.884013] Key type .llcrypt registered [ 5386.920806] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5387.311937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5387.328106] alg: No test for adler32 (adler32-zlib) [ 5388.178394] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5388.259150] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5389.839139] Key type lgssc registered [ 5390.183298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5392.725413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5392.732328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.879429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.116096] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5397.799325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5397.916232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5399.223867] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5401.102335] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5402.402465] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5404.752248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5404.882595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.910538] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_default_debug -1 all [ 5408.800836] Lustre: DEBUG MARKER: oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5428.706888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.710111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5428.712207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5428.714084] LustreError: Skipped 1 previous similar message [ 5428.717631] Lustre: Skipped 1 previous similar message [ 5433.826441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5433.828494] Lustre: Skipped 1 previous similar message [ 5434.219123] LustreError: 224876:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5434.277107] Lustre: server umount lustre-OST0000 complete [ 5438.945598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.945749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5438.947204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5438.948739] Lustre: Skipped 2 previous similar messages [ 5441.667846] LustreError: 225082:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5441.670825] LustreError: 225082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5441.701965] Lustre: server umount lustre-MDT0000 complete [ 5443.020417] LustreError: 224727:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758742432 with bad export cookie 13121511332486358682 [ 5443.023491] LustreError: MGC192.168.202.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5443.025145] LustreError: 224727:0:(ldlm_lockd.c:2607:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5443.095889] LustreError: 225284:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5443.097756] LustreError: 225284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5443.185372] Lustre: server umount lustre-MDT0001 complete [ 5446.977494] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5448.174366] Key type lgssc unregistered [ 5448.317447] LNet: 225816:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5448.320781] LNetError: 225816:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5448.329398] LNet: Removed LNI 192.168.202.145@tcp [ 5448.657188] Key type .llcrypt unregistered [ 5448.658406] Key type ._llcrypt unregistered [ 5456.192396] Key type ._llcrypt registered [ 5456.194430] Key type .llcrypt registered [ 5456.236386] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5456.356289] Key type .llcrypt unregistered [ 5456.357844] Key type ._llcrypt unregistered [ 5464.468274] Key type ._llcrypt registered [ 5464.469733] Key type .llcrypt registered [ 5464.509317] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing set_hostid [ 5470.726099] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing load_modules_local [ 5471.126133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5471.135974] alg: No test for adler32 (adler32-zlib) [ 5472.007085] Lustre: Lustre: Build Version: 2.16.58_53_g2e3f3ed [ 5472.094204] LNet: Added LNI 192.168.202.145@tcp [8/256/0/180] [ 5473.671149] Key type lgssc registered [ 5474.051389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5477.674484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5480.604792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5482.702570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5484.894172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5485.460439] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5237 sec ========== 15:34:34 (1758742474) [ 5486.037081] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:34:35 (1758742475) === [ 5487.145912] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:34:36 (1758742476) === [ 5499.590572] Lustre: DEBUG MARKER: oleg245-server.virtnet: executing unload_modules_local [ 5500.654158] Key type lgssc unregistered [ 5500.782383] LNet: 231891:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5500.785397] LNetError: 231891:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5500.797319] LNet: Removed LNI 192.168.202.145@tcp [ 5501.077354] Key type .llcrypt unregistered [ 5501.078827] Key type ._llcrypt unregistered