[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470523028 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003066] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.004901] kvm-guest: setup PV IPIs [ 0.006806] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007013] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010028] Yama: becoming mindful. [ 0.011022] SELinux: Initializing. [ 0.012043] *** VALIDATE selinux *** [ 0.019356] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023313] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024100] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025063] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026070] *** VALIDATE tmpfs *** [ 0.027180] *** VALIDATE proc *** [ 0.027981] *** VALIDATE cgroup *** [ 0.028004] *** VALIDATE cgroup2 *** [ 0.029212] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032017] Spectre V2 : User space: Vulnerable [ 0.033003] Speculative Store Bypass: Vulnerable [ 0.035915] debug: unmapping init [mem 0xffffffff85c59000-0xffffffff85c60fff] [ 0.037705] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038523] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039013] ... version: 2 [ 0.040007] ... bit width: 48 [ 0.041006] ... generic registers: 4 [ 0.041736] ... value mask: 0000ffffffffffff [ 0.042006] ... max period: 00007fffffffffff [ 0.043005] ... fixed-purpose events: 3 [ 0.043784] ... event mask: 000000070000000f [ 0.045055] rcu: Hierarchical SRCU implementation. [ 0.047548] smp: Bringing up secondary CPUs ... [ 0.048480] x86: Booting SMP configuration: [ 0.049012] .... node #0, CPUs: #1 #2 #3 [ 0.052095] smp: Brought up 1 node, 4 CPUs [ 0.054014] smpboot: Max logical packages: 1 [ 0.055012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233172] node 0 deferred pages initialised in 177ms [ 0.236031] devtmpfs: initialized [ 0.236879] x86/mm: Memory block size: 128MB [ 0.238380] gcov: version magic: 0x41383552 [ 0.240094] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.242105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243341] pinctrl core: initialized pinctrl subsystem [ 0.245087] [ 0.245576] ************************************************************* [ 0.246006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248007] ** ** [ 0.250008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.252009] ** ** [ 0.254007] ** This means that this kernel is built to expose internal ** [ 0.255006] ** IOMMU data structures, which may compromise security on ** [ 0.257007] ** your system. ** [ 0.258005] ** ** [ 0.259007] ** If you see this message and you are not debugging the ** [ 0.261006] ** kernel, report this immediately to your vendor! ** [ 0.262006] ** ** [ 0.263007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265008] ************************************************************* [ 0.266660] NET: Registered protocol family 16 [ 0.267321] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.273028] cpuidle: using governor menu [ 0.274415] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275094] PCI: Using configuration type 1 for base access [ 0.277099] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.284158] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.286024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288074] cryptd: max_cpu_qlen set to 1000 [ 0.290174] ACPI: Added _OSI(Module Device) [ 0.290902] ACPI: Added _OSI(Processor Device) [ 0.291007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.291878] ACPI: Added _OSI(Processor Aggregator Device) [ 0.294665] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.298382] ACPI: Interpreter enabled [ 0.299034] ACPI: PM: (supports S0 S3 S4 S5) [ 0.299777] ACPI: Using IOAPIC for interrupt routing [ 0.301047] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.303211] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.310136] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.311020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.313008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315035] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.317838] acpiphp: Slot [2] registered [ 0.318089] acpiphp: Slot [3] registered [ 0.318821] acpiphp: Slot [4] registered [ 0.320041] acpiphp: Slot [5] registered [ 0.320798] acpiphp: Slot [6] registered [ 0.321046] acpiphp: Slot [7] registered [ 0.321771] acpiphp: Slot [8] registered [ 0.323046] acpiphp: Slot [9] registered [ 0.323780] acpiphp: Slot [10] registered [ 0.324046] acpiphp: Slot [11] registered [ 0.324815] acpiphp: Slot [12] registered [ 0.326042] acpiphp: Slot [13] registered [ 0.326813] acpiphp: Slot [14] registered [ 0.327043] acpiphp: Slot [15] registered [ 0.327833] acpiphp: Slot [16] registered [ 0.329041] acpiphp: Slot [17] registered [ 0.329791] acpiphp: Slot [18] registered [ 0.330054] acpiphp: Slot [19] registered [ 0.330825] acpiphp: Slot [20] registered [ 0.332044] acpiphp: Slot [21] registered [ 0.332777] acpiphp: Slot [22] registered [ 0.333044] acpiphp: Slot [23] registered [ 0.333788] acpiphp: Slot [24] registered [ 0.335054] acpiphp: Slot [25] registered [ 0.335811] acpiphp: Slot [26] registered [ 0.336041] acpiphp: Slot [27] registered [ 0.336745] acpiphp: Slot [28] registered [ 0.338042] acpiphp: Slot [29] registered [ 0.338712] acpiphp: Slot [30] registered [ 0.339057] acpiphp: Slot [31] registered [ 0.339763] PCI host bridge to bus 0000:00 [ 0.340010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347012] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.349016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355534] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357718] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367968] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369009] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373542] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.375525] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.380489] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.384012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393016] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.397014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402591] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.408018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.427013] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.438151] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.464018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.473638] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.481013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.485014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.500030] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.509574] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.515014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.532020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.543013] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.555010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.574016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.582139] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.587009] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.592010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.610031] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.621978] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.624335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.626320] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.628291] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.630182] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.636042] iommu: Default domain type: Passthrough [ 0.638555] SCSI subsystem initialized [ 0.639109] ACPI: bus type USB registered [ 0.641139] usbcore: registered new interface driver usbfs [ 0.643069] usbcore: registered new interface driver hub [ 0.645069] usbcore: registered new device driver usb [ 0.647174] pps_core: LinuxPPS API ver. 1 registered [ 0.649010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.652057] PTP clock support registered [ 0.654146] EDAC MC: Ver: 3.0.0 [ 0.655566] PCI: Using ACPI for IRQ routing [ 0.657096] NetLabel: Initializing [ 0.658009] NetLabel: domain hash size = 128 [ 0.659011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.660173] NetLabel: unlabeled traffic allowed by default [ 0.662231] vgaarb: loaded [ 0.664356] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.666009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.672067] clocksource: Switched to clocksource kvm-clock [ 0.790256] VFS: Disk quotas dquot_6.6.0 [ 0.792240] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.795279] *** VALIDATE ramfs *** [ 0.796709] *** VALIDATE hugetlbfs *** [ 0.798395] pnp: PnP ACPI init [ 0.801068] pnp: PnP ACPI: found 6 devices [ 0.817518] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.821534] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.824189] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.826833] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.829644] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.832500] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.835424] NET: Registered protocol family 2 [ 0.838229] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.844201] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.847963] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.853478] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.856985] TCP: Hash tables configured (established 65536 bind 65536) [ 0.859857] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.862926] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.865941] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869078] NET: Registered protocol family 1 [ 0.871411] RPC: Registered named UNIX socket transport module. [ 0.873453] RPC: Registered udp transport module. [ 0.875122] RPC: Registered tcp transport module. [ 0.876814] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.879079] NET: Registered protocol family 44 [ 0.880672] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.882657] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.883833] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.885401] PCI: CLS 0 bytes, default 64 [ 0.886568] Unpacking initramfs... [ 2.260644] debug: unmapping init [mem 0xffff99f47cc54000-0xffff99f47ffbffff] [ 2.262772] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.264051] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.265577] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.732131] Initialise system trusted keyrings [ 2.733685] Key type blacklist registered [ 2.736152] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.746378] zbud: loaded [ 2.749354] *** VALIDATE nfs *** [ 2.751033] *** VALIDATE nfs4 *** [ 2.752517] pstore: using deflate compression [ 2.755883] Platform Keyring initialized [ 2.854787] NET: Registered protocol family 38 [ 2.856268] Key type asymmetric registered [ 2.857336] Asymmetric key parser 'x509' registered [ 2.858658] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.860758] io scheduler mq-deadline registered [ 2.862813] io scheduler kyber registered [ 2.864261] io scheduler bfq registered [ 2.865797] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.868846] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.871174] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.873641] ACPI: Power Button [PWRF] [ 2.965103] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.047056] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.256979] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.350520] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.532799] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.562695] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.591390] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.596246] Non-volatile memory driver v1.3 [ 3.597684] Linux agpgart interface v0.103 [ 3.626603] virtio_blk virtio1: [vda] 133248 512-byte logical blocks (68.2 MB/65.1 MiB) [ 3.629740] vda: detected capacity change from 0 to 68222976 [ 3.644290] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.645836] vdb: detected capacity change from 0 to 1073741824 [ 3.656065] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.657560] vdc: detected capacity change from 0 to 2621440000 [ 3.670172] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.673052] vdd: detected capacity change from 0 to 2621440000 [ 3.682561] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.684133] vde: detected capacity change from 0 to 4294967296 [ 3.700903] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.703425] vdf: detected capacity change from 0 to 4294967296 [ 3.710527] libphy: Fixed MDIO Bus: probed [ 3.717843] usbcore: registered new interface driver usbserial_generic [ 3.719879] usbserial: USB Serial support registered for generic [ 3.722103] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.727521] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.731158] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.733139] mousedev: PS/2 mouse device common for all mice [ 3.736484] rtc_cmos 00:05: RTC can wake from S4 [ 3.739622] rtc_cmos 00:05: registered as rtc0 [ 3.741441] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.745858] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.749727] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.752388] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.754882] intel_pstate: CPU model not supported [ 3.757283] hid: raw HID events driver (C) Jiri Kosina [ 3.759168] usbcore: registered new interface driver usbhid [ 3.761198] usbhid: USB HID core driver [ 3.762611] drop_monitor: Initializing network drop monitor service [ 3.765049] Initializing XFRM netlink socket [ 3.767129] NET: Registered protocol family 10 [ 3.770111] Segment Routing with IPv6 [ 3.771406] NET: Registered protocol family 17 [ 3.773364] mpls_gso: MPLS GSO support [ 3.778152] RAS: Correctable Errors collector initialized. [ 3.779740] AVX version of gcm_enc/dec engaged. [ 3.780931] AES CTR mode by8 optimization enabled [ 3.869893] sched_clock: Marking stable (3869875501, 0)->(4593731546, -723856045) [ 3.874160] registered taskstats version 1 [ 3.876917] Loading compiled-in X.509 certificates [ 3.879225] zswap: loaded using pool lzo/zbud [ 3.902436] Key type big_key registered [ 3.916316] Key type encrypted registered [ 3.917976] ima: No TPM chip found, activating TPM-bypass! [ 3.919633] ima: Allocated hash algorithm: sha1 [ 3.920940] ima: No architecture policies found [ 3.922243] evm: Initialising EVM extended attributes: [ 3.923687] evm: security.selinux [ 3.924668] evm: security.ima [ 3.925559] evm: security.capability [ 3.926604] evm: HMAC attrs: 0x1 [ 3.928572] rtc_cmos 00:05: setting system clock to 2025-10-10 23:06:06 UTC (1760137566) [ 3.934373] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 3.936795] debug: unmapping init [mem 0xffffffff85982000-0xffffffff85c58fff] [ 3.945262] Write protecting the kernel read-only data: 28672k [ 3.949782] debug: unmapping init [mem 0xffffffff84003000-0xffffffff841fffff] [ 3.953079] debug: unmapping init [mem 0xffffffff84914000-0xffffffff849fffff] [ 3.991619] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.998637] systemd[1]: Detected virtualization kvm. [ 4.000056] systemd[1]: Detected architecture x86-64. [ 4.001570] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.028424] systemd[1]: No hostname configured. [ 4.030258] systemd[1]: Set hostname to . [ 4.032601] random: systemd: uninitialized urandom read (16 bytes read) [ 4.035053] systemd[1]: Initializing machine ID from random generator. [ 4.077764] random: ln: uninitialized urandom read (6 bytes read) [ 4.165313] random: systemd: uninitialized urandom read (16 bytes read) [ 4.168128] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.172686] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.176918] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.843141] device-mapper: uevent: version 1.0.3 [ 4.846377] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.778689] virtio_net virtio0 ens2: renamed from eth0 [ 5.833191] random: fast init done [ 5.908738] scsi host0: ata_piix [ 5.931417] scsi host1: ata_piix [ 5.966514] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.968854] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.435515] random: crng init done [ 10.436666] random: 7 urandom warning(s) missed due to ratelimiting [ 10.537175] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.937882] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.213466] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.551658] SELinux: Disabled at runtime. [ 13.622254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.631101] systemd[1]: Detected virtualization kvm. [ 13.632785] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.180310] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.183787] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.189842] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.194388] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.197671] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.209812] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.219595] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 14.309853] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.769718] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.062462] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.071710] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.243950] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.259327] EDAC sbridge: Ver: 1.1.2 [ 18.021242] Key type dns_resolver registered [ 18.369627] NFS: Registering the id_resolver key type [ 18.371541] Key type id_resolver registered [ 18.373483] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg230-server login: [ 41.266210] libcfs: loading out-of-tree module taints kernel. [ 41.281221] Key type ._llcrypt registered [ 41.283181] Key type .llcrypt registered [ 41.339230] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 48.283772] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 48.960401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.970090] alg: No test for adler32 (adler32-zlib) [ 49.956344] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 50.241643] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 51.847218] Key type lgssc registered [ 52.400340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.783626] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 59.683387] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.015775] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 65.311798] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 67.759183] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.808738] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 77.107428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.127627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 77.139066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 78.226940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 78.239227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 78.286448] Lustre: lustre-MDT0000: new disk, initializing [ 78.311465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 78.318214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 79.609560] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 84.733453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.762066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 84.787400] Lustre: 6484:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 84.797666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 84.800627] Lustre: Skipped 1 previous similar message [ 84.845217] Lustre: lustre-MDT0001: new disk, initializing [ 84.864116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 84.871559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 84.876462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 86.080346] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.406163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 91.495369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.518971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 91.611345] Lustre: lustre-OST0000: new disk, initializing [ 91.614335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 91.632360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.534225] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 96.753809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 96.757217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 96.787914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 99.846435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.875089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.914956] Lustre: lustre-OST0001: new disk, initializing [ 99.916993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 99.936244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.091150] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.829135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 102.833956] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 102.842667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 107.957721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 114.714445] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 121.845773] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing check_logdir /tmp/testlogs/ [ 123.189082] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing yml_node [ 124.551299] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 125.400621] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 126.265776] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 126.812625] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 10 19:08:09 EDT 2025 [ 132.732281] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 133.256877] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 146.399556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 146.401685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.404869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.959874] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.960038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.962905] Lustre: Skipped 2 previous similar messages [ 148.964663] Lustre: Skipped 2 previous similar messages [ 151.163752] LustreError: 12187:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 151.194528] Lustre: server umount lustre-MDT0000 complete [ 152.432056] LustreError: 6476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137715 with bad export cookie 5069256345421527288 [ 152.433634] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 152.437607] LustreError: 6476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 154.079660] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.079882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 154.082588] Lustre: Skipped 1 previous similar message [ 157.087608] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 157.089272] Lustre: Skipped 2 previous similar messages [ 158.586956] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 158.588936] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 158.667721] Lustre: server umount lustre-MDT0001 complete [ 159.946175] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 159.950247] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 159.977552] Lustre: server umount lustre-OST0000 complete [ 161.314457] Lustre: server umount lustre-OST0001 complete [ 163.475576] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 166.534592] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 170.383861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.626274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 174.622932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.766978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.074972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 179.097623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.183142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.196212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.240855] Lustre: lustre-MDT0000: new disk, initializing [ 179.267717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.277304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 180.568429] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 184.643041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.663801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 184.695900] Lustre: 14968:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 184.802091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.122266] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 189.281522] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 190.443168] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 192.745099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.771889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 192.870455] Lustre: lustre-OST0000: new disk, initializing [ 192.871655] Lustre: Skipped 1 previous similar message [ 192.873380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.875058] Lustre: Skipped 2 previous similar messages [ 192.909439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.911198] Lustre: Skipped 1 previous similar message [ 194.039302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.041935] Lustre: Skipped 1 previous similar message [ 194.043900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.086927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.933593] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 198.079705] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 199.486579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 199.580264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 200.948841] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 201.046749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 203.757168] LustreError: 17906:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 203.759132] LustreError: 17906:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 203.820196] Lustre: server umount lustre-OST0000 complete [ 205.279996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 205.283819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 205.288492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 209.377893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 214.496348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 214.498325] Lustre: Skipped 3 previous similar messages [ 214.500213] LustreError: 18106:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 214.501809] LustreError: 18106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 214.527615] Lustre: server umount lustre-MDT0000 complete [ 215.775674] LustreError: 14958:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137778 with bad export cookie 5069256345421528807 [ 215.779187] LustreError: 14958:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 215.780505] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 226.275410] LustreError: 18308:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 226.277423] LustreError: 18308:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 226.368464] Lustre: server umount lustre-MDT0001 complete [ 227.536889] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:09:50 (1760137790) [ 229.773050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.949715] LustreError: 18833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 229.957298] LustreError: 18833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 229.980409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.320104] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 234.313750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 235.904231] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 237.882828] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 239.169685] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.405155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 243.427140] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 245.232213] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 259.040030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 259.042300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.048387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 264.561132] LustreError: 21099:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 264.564891] LustreError: 21099:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 264.632789] Lustre: server umount lustre-OST0000 complete [ 270.305220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 270.308689] LustreError: Skipped 1 previous similar message [ 272.211702] Lustre: server umount lustre-MDT0000 complete [ 273.494405] LustreError: 19550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137836 with bad export cookie 5069256345421529689 [ 273.497879] LustreError: 19550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 273.498316] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.634881] Lustre: server umount lustre-MDT0001 complete [ 277.602994] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 278.594372] Key type lgssc unregistered [ 278.722349] LNet: 22032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.724915] LNetError: 22032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.735320] LNet: Removed LNI 192.168.202.130@tcp [ 279.085142] Key type .llcrypt unregistered [ 279.086213] Key type ._llcrypt unregistered [ 287.837314] Key type ._llcrypt registered [ 287.838260] Key type .llcrypt registered [ 287.872078] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 288.171453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 288.236521] alg: No test for adler32 (adler32-zlib) [ 289.079906] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 289.154463] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 290.735105] Key type lgssc registered [ 291.055282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 293.665756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 293.674761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.825942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 296.032636] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 298.972184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 299.120873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.480174] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 302.302505] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 303.429051] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 305.712905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 305.843654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 307.671702] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 309.433577] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 321.190494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 326.629488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.630157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 326.630981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 326.632782] Lustre: Skipped 1 previous similar message [ 331.116196] LustreError: 26110:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 331.169881] Lustre: server umount lustre-OST0000 complete [ 334.815853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 334.817731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.820844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.824104] Lustre: Skipped 1 previous similar message [ 336.864399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.864832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.492416] LustreError: 26312:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 338.494317] LustreError: 26312:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 338.516231] Lustre: server umount lustre-MDT0000 complete [ 339.744428] LustreError: 26314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137902 with bad export cookie 16497799404990240588 [ 339.747938] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.751151] LustreError: 26314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 339.819342] LustreError: 26516:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 339.820970] LustreError: 26516:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 339.918268] Lustre: server umount lustre-MDT0001 complete [ 343.636351] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 344.761586] Key type lgssc unregistered [ 344.888460] LNet: 27049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.892207] LNetError: 27049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.901455] LNet: Removed LNI 192.168.202.130@tcp [ 345.220115] Key type .llcrypt unregistered [ 345.221109] Key type ._llcrypt unregistered [ 353.865565] Key type ._llcrypt registered [ 353.867024] Key type .llcrypt registered [ 353.905794] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 354.268499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 354.325469] alg: No test for adler32 (adler32-zlib) [ 355.169013] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 355.247743] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 356.831173] Key type lgssc registered [ 357.208070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 359.783533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 359.789784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.938675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.228328] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 365.113741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 365.256786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.660319] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 368.433787] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 369.685056] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 374.187948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 374.298300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.179460] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 377.955504] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 386.598625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 390.623910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 390.625984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.629451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 391.649423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 391.649958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.650050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 395.757138] LustreError: 31026:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 395.831137] Lustre: server umount lustre-OST0000 complete [ 401.375815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.377836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.381682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 401.384256] Lustre: Skipped 1 previous similar message [ 403.323990] LustreError: 31228:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 403.325804] LustreError: 31228:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 403.361597] Lustre: server umount lustre-MDT0000 complete [ 404.592248] LustreError: 31229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760137967 with bad export cookie 11726391496272921099 [ 404.594916] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.595153] LustreError: 31229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 404.663144] LustreError: 31430:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 404.664745] LustreError: 31430:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 404.747946] Lustre: server umount lustre-MDT0001 complete [ 408.356178] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 409.422904] Key type lgssc unregistered [ 409.558409] LNet: 31962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.560758] LNetError: 31962:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.573300] LNet: Removed LNI 192.168.202.130@tcp [ 409.865604] Key type .llcrypt unregistered [ 409.866631] Key type ._llcrypt unregistered [ 418.840455] Key type ._llcrypt registered [ 418.841811] Key type .llcrypt registered [ 418.883772] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 419.398957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 419.412254] alg: No test for adler32 (adler32-zlib) [ 420.257610] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 420.342853] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 421.919138] Key type lgssc registered [ 422.271782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 425.043556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 425.051838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.211029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.514091] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 430.509506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 430.629564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.970936] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 433.776704] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 435.108904] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 437.599818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 437.726478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.777376] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 441.589314] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 452.074853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 461.281624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.284740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.289979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.304944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.306774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.307255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.838131] LustreError: 35596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 465.910328] Lustre: server umount lustre-OST0000 complete [ 467.424509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.425285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.432639] Lustre: Skipped 1 previous similar message [ 467.435567] Lustre: Skipped 1 previous similar message [ 472.544582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.546633] Lustre: Skipped 2 previous similar messages [ 473.340655] LustreError: 35799:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 473.342721] LustreError: 35799:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 473.370723] Lustre: server umount lustre-MDT0000 complete [ 474.640438] LustreError: 33311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138037 with bad export cookie 12749660423063539474 [ 474.643047] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.644273] LustreError: 33311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 474.706152] LustreError: 36001:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 474.707654] LustreError: 36001:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 474.807461] Lustre: server umount lustre-MDT0001 complete [ 479.084615] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 480.366671] Key type lgssc unregistered [ 480.540652] LNet: 36532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.545481] LNetError: 36532:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.557604] LNet: Removed LNI 192.168.202.130@tcp [ 481.048614] Key type .llcrypt unregistered [ 481.050415] Key type ._llcrypt unregistered [ 491.483428] Key type ._llcrypt registered [ 491.485096] Key type .llcrypt registered [ 491.532251] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 491.858507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 491.888067] alg: No test for adler32 (adler32-zlib) [ 492.742815] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 492.825044] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 494.407183] Key type lgssc registered [ 494.928424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 498.010053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 498.025070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.251396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.796199] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 503.870880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 504.000829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.449843] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 507.247715] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 508.648823] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 511.145121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 511.257987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.326150] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 515.519768] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 523.563442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 528.868250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.869844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.872386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.876239] Lustre: Skipped 1 previous similar message [ 531.255465] Lustre: lustre-OST0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 531.259097] Lustre: Skipped 1 previous similar message [ 533.986793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.992030] Lustre: Skipped 1 previous similar message [ 534.445771] LustreError: 40168:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 534.527362] Lustre: server umount lustre-OST0000 complete [ 548.230745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.231659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.233898] Lustre: Skipped 1 previous similar message [ 553.441064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.443423] Lustre: Skipped 2 previous similar messages [ 554.364585] LustreError: 40545:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 554.366613] LustreError: 40545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 554.396618] Lustre: server umount lustre-MDT0000 complete [ 555.901471] LustreError: 38620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138118 with bad export cookie 10811485231696476528 [ 555.904377] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.906049] LustreError: 38620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 570.335136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 570.374158] LustreError: 40746:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 570.377192] LustreError: 40746:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 570.454197] Lustre: server umount lustre-MDT0001 complete [ 574.680169] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 576.126669] Key type lgssc unregistered [ 576.279509] LNet: 41279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.282398] LNetError: 41279:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.295469] LNet: Removed LNI 192.168.202.130@tcp [ 576.689577] Key type .llcrypt unregistered [ 576.691512] Key type ._llcrypt unregistered [ 586.374413] Key type ._llcrypt registered [ 586.375917] Key type .llcrypt registered [ 586.428189] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 586.857523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 586.908136] alg: No test for adler32 (adler32-zlib) [ 587.758283] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 587.845318] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 589.423172] Key type lgssc registered [ 589.898679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 592.506649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 592.513344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.664212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.938799] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 598.148941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 598.334620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 599.936434] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 602.014699] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 603.392114] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 605.995595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 606.148244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.219104] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 610.168666] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 618.473122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 623.585525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.586894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.588812] Lustre: Skipped 2 previous similar messages [ 623.593992] Lustre: Skipped 2 previous similar messages [ 625.971477] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 628.706237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.708899] Lustre: Skipped 2 previous similar messages [ 629.517538] LustreError: 44916:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 629.551805] Lustre: server umount lustre-MDT0000 complete [ 631.084986] LustreError: 44591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.096680] LustreError: 44591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 631.338117] LustreError: 42630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138193 with bad export cookie 9832697209732844523 [ 631.346382] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.825414] LustreError: 42642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.825810] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.842733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 636.203316] LustreError: 42643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.503395] LustreError: 45118:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 637.506127] LustreError: 45118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 637.588524] Lustre: server umount lustre-MDT0001 complete [ 655.602353] LustreError: 45341:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 655.607614] LustreError: 45341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 655.715897] Lustre: server umount lustre-OST0000 complete [ 662.694404] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 663.987759] Key type lgssc unregistered [ 664.121298] LNet: 46178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.123736] LNetError: 46178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.134262] LNet: Removed LNI 192.168.202.130@tcp [ 664.400132] Key type .llcrypt unregistered [ 664.401292] Key type ._llcrypt unregistered [ 676.375160] Key type ._llcrypt registered [ 676.377677] Key type .llcrypt registered [ 676.449579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 676.904308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.964936] alg: No test for adler32 (adler32-zlib) [ 677.887409] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 678.011351] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 679.615204] Key type lgssc registered [ 680.012475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 683.521345] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 683.528485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 698.975371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 704.095203] LustreError: 47523:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 730.719274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.401907] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 735.662800] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 901.440908] LustreError: 48233:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 901.540497] Lustre: server umount lustre-OST0000 complete [ 908.938408] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 910.358807] Key type lgssc unregistered [ 910.549642] LNet: 49069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.554287] LNetError: 49069:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.563644] LNet: Removed LNI 192.168.202.130@tcp [ 911.066395] Key type .llcrypt unregistered [ 911.067323] Key type ._llcrypt unregistered [ 921.702551] Key type ._llcrypt registered [ 921.704127] Key type .llcrypt registered [ 921.746964] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 922.015573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 922.023898] alg: No test for adler32 (adler32-zlib) [ 922.882803] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 922.998943] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 924.591190] Key type lgssc registered [ 925.097428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 928.544466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 928.556727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.754856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.144201] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 934.266058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 934.409222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.914628] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 938.048184] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 939.638633] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 942.660220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 942.844255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.330217] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 947.328752] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 949.220701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.222098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 949.229377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 950.323193] LustreError: 52681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 950.428962] Lustre: server umount lustre-OST0000 complete [ 952.191695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.199693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 952.203914] Lustre: Skipped 1 previous similar message [ 957.408952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 957.412656] Lustre: Skipped 1 previous similar message [ 958.347936] LustreError: 52882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 958.351991] LustreError: 52882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 958.391206] Lustre: server umount lustre-MDT0000 complete [ 960.208952] LustreError: 50420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138522 with bad export cookie 5456511322063886472 [ 960.215153] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.216460] LustreError: 50420:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 960.314783] LustreError: 53085:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 960.318133] LustreError: 53085:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 960.419080] Lustre: server umount lustre-MDT0001 complete [ 965.285378] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 966.722654] Key type lgssc unregistered [ 966.912713] LNet: 53616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.917563] LNetError: 53616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.931551] LNet: Removed LNI 192.168.202.130@tcp [ 967.387170] Key type .llcrypt unregistered [ 967.389266] Key type ._llcrypt unregistered [ 979.089063] Key type ._llcrypt registered [ 979.090520] Key type .llcrypt registered [ 979.146818] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 979.546509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.599885] alg: No test for adler32 (adler32-zlib) [ 980.450665] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 980.538539] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 982.135222] Key type lgssc registered [ 982.784779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.754531] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.765878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1002.271492] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1007.391302] LustreError: 54961:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 1034.015162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.858362] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1041.336820] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1045.169180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.510100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.598441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1048.368693] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1053.521200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1053.695618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1055.263175] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1056.993956] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1058.121497] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1060.832192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1060.834824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.839382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.905812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.906394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.908290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.913330] Lustre: Skipped 1 previous similar message [ 1069.026747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.028430] Lustre: Skipped 1 previous similar message [ 1073.119145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1073.135350] LustreError: 57241:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1073.213077] Lustre: server umount lustre-OST0000 complete [ 1087.402204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.408079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1087.409554] Lustre: Skipped 1 previous similar message [ 1092.576285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.577883] Lustre: Skipped 1 previous similar message [ 1093.633967] LustreError: 57620:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1093.636753] LustreError: 57620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1093.668133] Lustre: server umount lustre-MDT0000 complete [ 1094.954518] LustreError: 54968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138657 with bad export cookie 11099549942633012139 [ 1094.956478] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1094.959369] LustreError: 54968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1101.057108] LustreError: 57822:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1101.060693] LustreError: 57822:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1101.154626] Lustre: server umount lustre-MDT0001 complete [ 1105.135802] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1106.337401] Key type lgssc unregistered [ 1106.504406] LNet: 58354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.507685] LNetError: 58354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.517399] LNet: Removed LNI 192.168.202.130@tcp [ 1106.855947] Key type .llcrypt unregistered [ 1106.857091] Key type ._llcrypt unregistered [ 1116.111872] Key type ._llcrypt registered [ 1116.113346] Key type .llcrypt registered [ 1116.154301] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1116.579995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1116.596940] alg: No test for adler32 (adler32-zlib) [ 1117.445398] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1117.526917] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1119.103182] Key type lgssc registered [ 1119.513519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1122.177734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.186184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.366836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.626934] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1127.631977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1127.763805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1129.083455] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1130.935644] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.189167] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1134.485199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1134.613171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.540912] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1138.395996] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1147.939342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1152.994220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1152.995496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1152.998230] LustreError: Skipped 1 previous similar message [ 1152.998906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1153.004379] Lustre: Skipped 1 previous similar message [ 1153.010029] Lustre: Skipped 1 previous similar message [ 1157.036175] LustreError: 61995:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1157.102159] Lustre: server umount lustre-OST0000 complete [ 1158.623854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1158.627566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.634335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.232510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.233327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.474801] LustreError: 62196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1164.478062] LustreError: 62196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1164.507646] Lustre: server umount lustre-MDT0000 complete [ 1165.730412] LustreError: 59703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138728 with bad export cookie 10236316228138077794 [ 1165.733720] LustreError: 59703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1165.733764] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.803917] LustreError: 62398:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1165.805549] LustreError: 62398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1165.883486] Lustre: server umount lustre-MDT0001 complete [ 1169.798555] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1170.893838] Key type lgssc unregistered [ 1171.037403] LNet: 62929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.039828] LNetError: 62929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.049454] LNet: Removed LNI 192.168.202.130@tcp [ 1171.364117] Key type .llcrypt unregistered [ 1171.365246] Key type ._llcrypt unregistered [ 1183.786829] Key type ._llcrypt registered [ 1183.788014] Key type .llcrypt registered [ 1183.828489] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1184.238644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1184.270703] alg: No test for adler32 (adler32-zlib) [ 1185.142504] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1185.227289] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1186.815221] Key type lgssc registered [ 1187.224178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1190.372888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1190.379830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.592103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.260515] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1197.553647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1197.765862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.612978] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1202.045551] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1203.834321] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1207.094314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1207.254894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.837460] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1212.205610] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1216.495429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1220.009208] Lustre: Failing over lustre-MDT0000 [ 1220.221525] LustreError: 66836:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1220.246228] Lustre: server umount lustre-MDT0000 complete [ 1221.604240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.605652] LustreError: 64591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.607639] Lustre: Skipped 2 previous similar messages [ 1221.611328] LustreError: 64591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1222.960629] LustreError: 64585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.967414] LustreError: 64585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1223.604879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.645926] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.703145] Lustre: *** cfs_fail_loc=135, val=0*** [ 1223.750354] LustreError: 67346:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1223.754848] LustreError: 67346:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.130@tcp: cfg command failed: rc = -2 [ 1223.760679] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1223.760679] [ 1223.764443] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1223.772480] LustreError: 67334:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1223.776198] LustreError: 67334:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1223.781271] LustreError: 67334:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1223.794519] Lustre: server umount lustre-MDT0000 complete [ 1223.796463] LustreError: 67334:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1226.313113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.363001] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1226.461991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.491516] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1228.075193] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1228.261020] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1231.857065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1231.874423] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1231.911644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1233.888553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1233.893134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.901101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.967352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.967702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.976119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.154200] LustreError: 68153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1239.158036] LustreError: 68153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1239.261248] Lustre: server umount lustre-OST0000 complete [ 1242.080859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1242.080895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.090553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.092155] Lustre: Skipped 2 previous similar messages [ 1247.201503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.205097] Lustre: Skipped 1 previous similar message [ 1252.321061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1252.323956] Lustre: Skipped 1 previous similar message [ 1255.391207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1255.450715] LustreError: 68354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1255.454327] LustreError: 68354:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1255.483050] Lustre: server umount lustre-MDT0000 complete [ 1257.265735] LustreError: 64572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138819 with bad export cookie 14201611603887343596 [ 1257.270112] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.272046] LustreError: 64572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1257.449584] Lustre: server umount lustre-MDT0001 complete [ 1261.525535] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1262.484049] Key type lgssc unregistered [ 1262.611378] LNet: 69087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.613958] LNetError: 69087:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.625274] LNet: Removed LNI 192.168.202.130@tcp [ 1262.948927] Key type .llcrypt unregistered [ 1262.950471] Key type ._llcrypt unregistered [ 1272.514019] Key type ._llcrypt registered [ 1272.515336] Key type .llcrypt registered [ 1272.554224] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1272.871030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1272.881921] alg: No test for adler32 (adler32-zlib) [ 1273.736563] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1273.830939] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1275.423200] Key type lgssc registered [ 1275.981839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1279.134105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1279.146870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.341908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.948537] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1285.843338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1286.000058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1287.692320] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1290.052483] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.380696] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1293.647657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1293.822869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.183931] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1298.310308] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1305.132919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1311.504071] Lustre: Failing over lustre-MDT0000 [ 1311.712502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.716353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.720576] LustreError: 72724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1311.722943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.758746] Lustre: server umount lustre-MDT0000 complete [ 1314.103241] LustreError: 70452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.113478] LustreError: 70452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1315.300207] LustreError: 71184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.300263] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.082991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.143279] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.213939] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1316.280080] LustreError: 73232:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1316.282382] LustreError: 73232:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.130@tcp: cfg command failed: rc = -9 [ 1316.285208] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1316.285208] [ 1316.288214] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1316.293226] LustreError: 73220:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1316.296116] LustreError: 73220:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1316.299311] LustreError: 73220:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1316.308822] Lustre: server umount lustre-MDT0000 complete [ 1316.310072] LustreError: 73220:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1318.651526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.724892] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.866326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.899224] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1319.212292] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1320.433651] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1324.023424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1324.033275] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1324.063133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1326.560338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1326.563765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.569468] Lustre: Skipped 1 previous similar message [ 1326.572394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1331.377170] LustreError: 74042:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1331.380800] LustreError: 74042:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1331.467503] Lustre: server umount lustre-OST0000 complete [ 1334.240114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.246154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.253834] Lustre: Skipped 1 previous similar message [ 1334.257936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.260713] Lustre: Skipped 2 previous similar messages [ 1337.312343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1337.316858] Lustre: Skipped 1 previous similar message [ 1339.203823] LustreError: 74245:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1339.207825] LustreError: 74245:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1339.239588] Lustre: server umount lustre-MDT0000 complete [ 1340.815990] LustreError: 74043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138903 with bad export cookie 15562289281401405501 [ 1340.822813] LustreError: 74043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1340.822845] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1341.003630] Lustre: server umount lustre-MDT0001 complete [ 1345.353636] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1346.882364] Key type lgssc unregistered [ 1347.038713] LNet: 74979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.043545] LNetError: 74979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.052610] LNet: Removed LNI 192.168.202.130@tcp [ 1347.543816] Key type .llcrypt unregistered [ 1347.545957] Key type ._llcrypt unregistered [ 1352.541449] Key type ._llcrypt registered [ 1352.543346] Key type .llcrypt registered [ 1352.593893] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1352.764522] Key type .llcrypt unregistered [ 1352.766849] Key type ._llcrypt unregistered [ 1364.294768] Key type ._llcrypt registered [ 1364.296266] Key type .llcrypt registered [ 1364.341023] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1364.638374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.648851] alg: No test for adler32 (adler32-zlib) [ 1365.496238] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1365.577341] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1367.151239] Key type lgssc registered [ 1367.689133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1371.003158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1371.015598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.205587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.572514] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1377.182064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1377.396169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1378.965330] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1381.199615] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.698049] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1385.682318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1385.874889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.458931] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1390.861909] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1397.162975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1402.336747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1402.337507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.339220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.341352] LustreError: Skipped 1 previous similar message [ 1402.345775] Lustre: Skipped 1 previous similar message [ 1406.705114] LustreError: 79066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1406.799291] Lustre: server umount lustre-OST0000 complete [ 1412.577619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.579106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.584826] Lustre: Skipped 1 previous similar message [ 1412.588419] Lustre: Skipped 1 previous similar message [ 1414.525644] LustreError: 79268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1414.527610] LustreError: 79268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1414.555802] Lustre: server umount lustre-MDT0000 complete [ 1415.979903] LustreError: 76799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760138978 with bad export cookie 16875700323421894162 [ 1415.987142] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1415.990935] LustreError: 76799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1416.085341] LustreError: 79470:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1416.087906] LustreError: 79470:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1416.173341] Lustre: server umount lustre-MDT0001 complete [ 1420.610516] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1422.316877] Key type lgssc unregistered [ 1422.488820] LNet: 80001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.495197] LNetError: 80001:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.505654] LNet: Removed LNI 192.168.202.130@tcp [ 1423.017912] Key type .llcrypt unregistered [ 1423.021233] Key type ._llcrypt unregistered [ 1433.931181] Key type ._llcrypt registered [ 1433.932090] Key type .llcrypt registered [ 1433.972741] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1434.418614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.475607] alg: No test for adler32 (adler32-zlib) [ 1435.334713] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1435.416674] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1436.991196] Key type lgssc registered [ 1437.616469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.727457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1441.740924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.971378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1444.552315] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1448.549477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1448.742080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1450.315669] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1452.848577] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1454.457376] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1457.593900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1457.791968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.442213] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1462.591310] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1468.076199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1473.507394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1473.510356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.513598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.520046] Lustre: Skipped 1 previous similar message [ 1476.720192] LustreError: 83618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1476.814409] Lustre: server umount lustre-OST0000 complete [ 1478.625648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.626852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.636761] Lustre: Skipped 1 previous similar message [ 1478.640810] Lustre: Skipped 1 previous similar message [ 1483.745890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1483.749445] Lustre: Skipped 2 previous similar messages [ 1484.752230] LustreError: 83819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1484.756596] LustreError: 83819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1484.794084] Lustre: server umount lustre-MDT0000 complete [ 1486.391823] LustreError: 81352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139048 with bad export cookie 5129426216935137681 [ 1486.398170] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.398693] LustreError: 81352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1486.491080] LustreError: 84021:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1486.493750] LustreError: 84021:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1486.580425] Lustre: server umount lustre-MDT0001 complete [ 1490.915150] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1492.286228] Key type lgssc unregistered [ 1492.406325] LNet: 84553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.408459] LNetError: 84553:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.417300] LNet: Removed LNI 192.168.202.130@tcp [ 1492.794465] Key type .llcrypt unregistered [ 1492.795569] Key type ._llcrypt unregistered [ 1502.319096] Key type ._llcrypt registered [ 1502.321378] Key type .llcrypt registered [ 1502.373533] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1502.703699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.746190] alg: No test for adler32 (adler32-zlib) [ 1503.601380] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1503.678351] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1505.263226] Key type lgssc registered [ 1505.906143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1509.414323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1509.420822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.580043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.138259] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1515.924385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1516.079663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1517.698469] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1519.998910] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1521.545212] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1524.664517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1524.837476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.300686] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1529.676919] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1537.131243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1546.721483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1546.725979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.734289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1547.746563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1547.750918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1547.751244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.153796] LustreError: 88189:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1551.249892] Lustre: server umount lustre-OST0000 complete [ 1552.865440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.873614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.876641] Lustre: Skipped 2 previous similar messages [ 1557.984692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1557.987771] Lustre: Skipped 1 previous similar message [ 1563.104537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.107251] Lustre: Skipped 1 previous similar message [ 1567.199218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1567.249395] LustreError: 88391:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1567.253460] LustreError: 88391:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1567.288587] Lustre: server umount lustre-MDT0000 complete [ 1568.224910] LustreError: 85923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.234429] LustreError: 85923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1568.881878] LustreError: 85905:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139131 with bad export cookie 15041317878272950002 [ 1568.887391] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.963507] LustreError: 88595:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1568.965409] LustreError: 88595:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1569.065471] Lustre: server umount lustre-MDT0001 complete [ 1573.627542] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1575.123827] Key type lgssc unregistered [ 1575.324731] LNet: 89126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.328197] LNetError: 89126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.340582] LNet: Removed LNI 192.168.202.130@tcp [ 1575.797315] Key type .llcrypt unregistered [ 1575.799756] Key type ._llcrypt unregistered [ 1585.921150] Key type ._llcrypt registered [ 1585.923614] Key type .llcrypt registered [ 1585.976805] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1586.333871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.340450] alg: No test for adler32 (adler32-zlib) [ 1587.214197] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1587.328245] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1588.911238] Key type lgssc registered [ 1589.515504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1593.245399] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1593.253266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1608.735463] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1613.855299] LustreError: 90481:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 1640.479273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.985277] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1645.481967] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1647.920859] LustreError: 91338:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1647.987187] Lustre: server umount lustre-OST0000 complete [ 1651.393153] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:33:33 (1760139213) [ 1654.047031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.218070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.812878] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1659.064858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1659.189505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1660.508726] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1662.871379] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1664.535718] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1667.416317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1667.570802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.134097] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1672.212141] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1678.831637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1689.061965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.062525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1689.066461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1689.067013] LustreError: Skipped 1 previous similar message [ 1689.068516] Lustre: Skipped 1 previous similar message [ 1692.785934] LustreError: 94431:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1692.787926] LustreError: 94431:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1692.887425] Lustre: server umount lustre-OST0000 complete [ 1695.200036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1695.202135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.205343] Lustre: Skipped 1 previous similar message [ 1695.206968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.298875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.299455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.416319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.419161] Lustre: Skipped 2 previous similar messages [ 1708.512397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1708.514081] Lustre: Skipped 1 previous similar message [ 1709.023124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1709.061499] LustreError: 94634:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1709.064171] LustreError: 94634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1709.097013] Lustre: server umount lustre-MDT0000 complete [ 1710.786480] LustreError: 91987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139273 with bad export cookie 3233745106714096892 [ 1710.791977] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1710.793311] LustreError: 91987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1710.966328] Lustre: server umount lustre-MDT0001 complete [ 1715.445851] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1716.780220] Key type lgssc unregistered [ 1716.904339] LNet: 95367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.906594] LNetError: 95367:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.917267] LNet: Removed LNI 192.168.202.130@tcp [ 1717.218347] Key type .llcrypt unregistered [ 1717.219534] Key type ._llcrypt unregistered [ 1727.667532] Key type ._llcrypt registered [ 1727.668751] Key type .llcrypt registered [ 1727.706333] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1728.025801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.041073] alg: No test for adler32 (adler32-zlib) [ 1728.912388] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1728.998761] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1730.575220] Key type lgssc registered [ 1731.050757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1733.929111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1733.936801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1735.111149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.757233] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1740.862573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1741.031409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.830428] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1745.318955] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1746.770345] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1749.133427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1749.268945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1751.223271] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1753.042491] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1759.529394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1766.880745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.884107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.890128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1769.956198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.956892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1769.960969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1772.780116] LustreError: 99005:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1772.850163] Lustre: server umount lustre-OST0000 complete [ 1775.072594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.076284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.077806] Lustre: Skipped 2 previous similar messages [ 1780.193195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.197440] Lustre: Skipped 1 previous similar message [ 1780.492568] LustreError: 99206:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1780.496630] LustreError: 99206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1780.530284] Lustre: server umount lustre-MDT0000 complete [ 1782.254627] LustreError: 98357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139344 with bad export cookie 10037735607282729602 [ 1782.261522] LustreError: 98357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1782.261788] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1782.350781] LustreError: 99409:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1782.353851] LustreError: 99409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1782.436812] Lustre: server umount lustre-MDT0001 complete [ 1786.601966] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1787.781512] Key type lgssc unregistered [ 1787.935663] LNet: 99940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.940348] LNetError: 99940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.948591] LNet: Removed LNI 192.168.202.130@tcp [ 1788.455813] Key type .llcrypt unregistered [ 1788.457943] Key type ._llcrypt unregistered [ 1796.705785] Key type ._llcrypt registered [ 1796.707509] Key type .llcrypt registered [ 1796.770816] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1797.185461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.213098] alg: No test for adler32 (adler32-zlib) [ 1798.110659] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 1798.220203] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1799.831158] Key type lgssc registered [ 1800.436604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.172622] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.182570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1819.679398] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1824.799351] LustreError: 101139:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 1851.423299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.892541] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1856.125001] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1859.034994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.131726] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1859.136043] LustreError: 102050:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1859.138371] LustreError: 102050:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1859.142328] LustreError: 102050:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1859.200493] Lustre: server umount lustre-MDT0000 complete [ 1859.202038] LustreError: 102050:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1873.375502] LustreError: 102481:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1873.440813] Lustre: server umount lustre-OST0000 complete [ 1877.286745] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 1881.258241] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1885.399651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1888.606134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1891.549979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1893.911687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1896.739991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1896.768311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.863554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1896.877427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1896.916166] Lustre: lustre-MDT0000: new disk, initializing [ 1896.938932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.945621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1898.196841] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1903.062715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1903.100672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1903.157107] Lustre: 104816:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1903.200284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1903.203840] Lustre: Skipped 1 previous similar message [ 1903.249720] Lustre: lustre-MDT0001: new disk, initializing [ 1903.285076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.299074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1903.303777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1904.859312] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1908.606850] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1910.046199] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1913.177160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1913.206586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1913.352619] Lustre: lustre-OST0000: new disk, initializing [ 1913.356334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1913.401475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.543377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1913.547135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1913.580989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1915.712551] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1919.442586] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1921.292054] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1921.402381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1923.205055] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1923.319300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1929.186261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.187821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.191280] LustreError: Skipped 1 previous similar message [ 1929.199937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.204964] Lustre: Skipped 1 previous similar message [ 1930.350075] LustreError: 107753:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1930.352223] LustreError: 107753:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1930.422593] Lustre: server umount lustre-OST0000 complete [ 1934.306424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1934.306788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.318827] Lustre: Skipped 1 previous similar message [ 1934.323062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.326360] Lustre: Skipped 1 previous similar message [ 1938.304612] LustreError: 107954:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1938.307120] LustreError: 107954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1938.336419] Lustre: server umount lustre-MDT0000 complete [ 1939.915967] LustreError: 104806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139502 with bad export cookie 8687681795436438891 [ 1939.919759] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.922536] LustreError: 104806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1940.101606] Lustre: server umount lustre-MDT0001 complete [ 1943.851324] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:38:26 (1760139506) [ 1945.423703] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1952.518725] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 1955.708677] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1959.672054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1962.621858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1964.660054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1967.068075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1970.163496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1970.205794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1970.322818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1970.341368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1970.388355] Lustre: lustre-MDT0000: new disk, initializing [ 1970.425459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.437177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1971.927461] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1976.987061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1977.036366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1977.081199] Lustre: 111286:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.115125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1977.118024] Lustre: Skipped 1 previous similar message [ 1977.162841] Lustre: lustre-MDT0001: new disk, initializing [ 1977.221893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1977.226391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1978.709389] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1982.427882] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.992949] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1987.248994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1987.285084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1987.436845] Lustre: lustre-OST0000: new disk, initializing [ 1987.440645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1987.484763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.489129] Lustre: Skipped 1 previous similar message [ 1989.767725] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1992.720434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1992.729051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1992.781765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1992.915272] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1994.461118] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1994.546177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1995.944580] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1996.021646] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1997.798128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.800475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1997.805155] Lustre: Skipped 1 previous similar message [ 1997.806253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1997.809487] LustreError: Skipped 1 previous similar message [ 1997.817867] Lustre: Skipped 1 previous similar message [ 1998.962186] LustreError: 114225:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1998.966179] LustreError: 114225:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1999.055544] Lustre: server umount lustre-OST0000 complete [ 2002.913529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.913728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2002.918163] Lustre: Skipped 1 previous similar message [ 2002.918756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.924567] Lustre: Skipped 1 previous similar message [ 2006.590006] LustreError: 114426:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2006.593197] LustreError: 114426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2006.626881] Lustre: server umount lustre-MDT0000 complete [ 2008.033077] LustreError: 111297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.042440] LustreError: 111297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2008.286107] LustreError: 111278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139570 with bad export cookie 8687681795436439759 [ 2008.291680] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2008.292267] LustreError: 111278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2008.473627] Lustre: server umount lustre-MDT0001 complete [ 2011.839400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.075839] LustreError: 115102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.121210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.717620] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2017.084089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2018.545933] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2020.739288] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2022.349944] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2024.789771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2027.538963] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2029.792146] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2043.097255] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2047.458830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.460440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.463365] LustreError: Skipped 1 previous similar message [ 2047.470322] Lustre: Skipped 1 previous similar message [ 2047.471634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2047.478816] Lustre: Skipped 1 previous similar message [ 2050.157193] LustreError: 117467:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2050.158975] LustreError: 117467:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2050.213078] Lustre: server umount lustre-OST0000 complete [ 2057.696954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.700693] Lustre: Skipped 3 previous similar messages [ 2057.840717] Lustre: server umount lustre-MDT0000 complete [ 2059.431221] LustreError: 115081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139622 with bad export cookie 8687681795436440634 [ 2059.434328] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.434480] LustreError: 115081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2059.569073] Lustre: server umount lustre-MDT0001 complete [ 2063.369780] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2064.722945] Key type lgssc unregistered [ 2064.884655] LNet: 118403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.889309] LNetError: 118403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.898565] LNet: Removed LNI 192.168.202.130@tcp [ 2065.346080] Key type .llcrypt unregistered [ 2065.347794] Key type ._llcrypt unregistered [ 2072.145872] Key type ._llcrypt registered [ 2072.147331] Key type .llcrypt registered [ 2072.197593] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2078.953897] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2079.297511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2079.328155] alg: No test for adler32 (adler32-zlib) [ 2080.199166] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2080.289481] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2081.879254] Key type lgssc registered [ 2082.426642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2086.181821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2088.724230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2091.277389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2093.783633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2096.768234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2096.803469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2096.815978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.904685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2097.915543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2097.950373] Lustre: lustre-MDT0000: new disk, initializing [ 2097.973937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.979853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2099.201169] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2104.201379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.230917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2104.284588] Lustre: 121504:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2104.315849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2104.318731] Lustre: Skipped 1 previous similar message [ 2104.357778] Lustre: lustre-MDT0001: new disk, initializing [ 2104.386331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2104.396399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2104.399378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2105.844788] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2109.570684] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.987151] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2114.155490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2114.190311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2114.298559] Lustre: lustre-OST0000: new disk, initializing [ 2114.301446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2114.338334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.725533] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2119.696314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2119.705928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2119.760553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2120.677932] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2122.480285] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2122.615403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2124.350485] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2124.447487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2129.892254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.892338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.895829] LustreError: Skipped 1 previous similar message [ 2129.902277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2129.906328] Lustre: Skipped 1 previous similar message [ 2131.378173] LustreError: 124442:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2131.467822] Lustre: server umount lustre-OST0000 complete [ 2135.010847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2135.011092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.022369] Lustre: Skipped 1 previous similar message [ 2135.026590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.031406] Lustre: Skipped 1 previous similar message [ 2139.339456] LustreError: 124643:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2139.343521] LustreError: 124643:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2139.383053] Lustre: server umount lustre-MDT0000 complete [ 2141.113302] LustreError: 122283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139703 with bad export cookie 75428884057814611 [ 2141.119548] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2141.120042] LustreError: 122283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2141.208923] LustreError: 124845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2141.212266] LustreError: 124845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2141.297974] Lustre: server umount lustre-MDT0001 complete [ 2144.845381] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:41:47 (1760139707) [ 2147.887024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.127529] LustreError: 125517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.137098] LustreError: 125517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2148.168531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.956153] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2153.442206] LustreError: 125517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.988082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2154.171342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2155.975877] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2158.529821] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2160.193498] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2163.682637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.683660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.686979] Lustre: Skipped 2 previous similar messages [ 2167.366133] LustreError: 126858:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2167.369580] LustreError: 126858:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2167.402396] Lustre: server umount lustre-MDT0000 complete [ 2168.911311] LustreError: 125497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139731 with bad export cookie 75428884057815479 [ 2168.915670] LustreError: 125497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2168.915688] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2169.066734] Lustre: server umount lustre-MDT0001 complete [ 2172.501279] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:42:14 (1760139734) [ 2175.155963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2190.623668] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2195.743510] LustreError: 127704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2221.344035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.175651] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2226.590547] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2227.565217] LustreError: 128368:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2227.569082] LustreError: 128368:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2227.670574] Lustre: server umount lustre-OST0000 complete [ 2231.272687] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:43:13 (1760139793) [ 2234.343644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.520976] LustreError: 129037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.243378] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2239.969597] LustreError: 129037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.091661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2240.280612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2240.282719] Lustre: Skipped 1 previous similar message [ 2242.220804] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2244.974778] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2246.689119] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2249.803339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2252.471279] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2254.996599] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2270.691006] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.693077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.697352] Lustre: Skipped 2 previous similar messages [ 2270.701624] Lustre: Skipped 4 previous similar messages [ 2274.112645] LustreError: 131305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2274.115824] LustreError: 131305:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2274.145900] Lustre: server umount lustre-MDT0000 complete [ 2275.470639] LustreError: 129019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139838 with bad export cookie 75428884057816130 [ 2275.474468] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2275.477160] LustreError: 129019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2275.664316] Lustre: server umount lustre-MDT0001 complete [ 2287.060453] Lustre: server umount lustre-OST0000 complete [ 2290.304764] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:44:12 (1760139852) [ 2292.949828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.151425] LustreError: 132379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.172979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.174774] Lustre: Skipped 1 previous similar message [ 2294.677300] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2298.465150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2300.280634] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2302.865810] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2304.403048] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2307.602449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2310.473792] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2313.093677] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2315.135571] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2318.066965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2318.387420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2320.323325] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2320.431207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2323.429167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.429914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.437637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.444250] Lustre: Skipped 1 previous similar message [ 2327.405952] LustreError: 135145:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2327.409137] LustreError: 135145:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2327.480546] Lustre: server umount lustre-OST0000 complete [ 2329.568334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.664689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.665487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.668549] Lustre: Skipped 1 previous similar message [ 2333.674495] Lustre: Skipped 2 previous similar messages [ 2335.402921] Lustre: server umount lustre-MDT0000 complete [ 2336.797063] LustreError: 132360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139899 with bad export cookie 75428884057817138 [ 2336.803542] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2336.804099] LustreError: 132360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2336.973429] Lustre: server umount lustre-MDT0001 complete [ 2340.209811] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:45:02 (1760139902) [ 2343.278185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2358.751599] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2363.871576] LustreError: 136189:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2389.471396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.475397] Lustre: Skipped 2 previous similar messages [ 2391.497978] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2393.601668] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2396.650239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.811692] LustreError: 136216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.819155] LustreError: 136216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2396.864339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2398.191036] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2401.491991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2403.121626] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2405.328521] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.696863] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2408.135611] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2408.200091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2409.706796] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2409.805955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2412.000982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2412.005438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.012671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.311154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2425.362449] LustreError: 138967:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2425.366326] LustreError: 138967:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2425.397837] Lustre: server umount lustre-MDT0000 complete [ 2427.073908] LustreError: 137826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760139989 with bad export cookie 75428884057818153 [ 2427.080299] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.080486] LustreError: 137826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2427.239690] Lustre: server umount lustre-MDT0001 complete [ 2439.082702] Lustre: server umount lustre-OST0000 complete [ 2442.955091] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:46:45 (1760140005) [ 2445.962189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2461.407725] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2466.527609] LustreError: 140016:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2494.699310] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2497.407833] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2500.603684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.777576] LustreError: 140043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.825994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2502.219122] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2505.416293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2507.295256] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2509.999612] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2511.877892] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2515.150411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.191964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2515.273512] Lustre: lustre-OST0001: new disk, initializing [ 2515.275763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2517.307102] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2521.277360] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2522.626771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2522.631653] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2522.678867] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2523.070236] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2523.152746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2524.764943] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2524.830716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2526.176742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2526.178636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.182201] Lustre: Skipped 2 previous similar messages [ 2526.183708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2526.185189] Lustre: Skipped 9 previous similar messages [ 2531.442746] LustreError: 143744:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2531.446443] LustreError: 143744:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2531.471661] Lustre: server umount lustre-OST0000 complete [ 2532.833600] LustreError: 141670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2532.838566] LustreError: 141670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2547.167088] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2547.239717] Lustre: server umount lustre-OST0001 complete [ 2551.775895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2551.777820] LustreError: Skipped 3 previous similar messages [ 2554.515840] Lustre: server umount lustre-MDT0000 complete [ 2555.670048] LustreError: 140024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140118 with bad export cookie 75428884057819098 [ 2555.673067] LustreError: 140024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2555.673972] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.810115] Lustre: server umount lustre-MDT0001 complete [ 2557.881661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2559.136293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2560.312180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2561.440683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2563.457415] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:48:45 (1760140125) [ 2563.975793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2564.482438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2564.988900] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:48:47 (1760140127) [ 2567.025530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.118880] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2567.129618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2567.197491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2567.199269] Lustre: Skipped 6 previous similar messages [ 2568.372909] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2570.972979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2571.001638] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2571.012687] Lustre: 145984:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2572.281633] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2573.954447] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2575.099940] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2577.162715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2577.229770] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2578.931925] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2580.579282] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2581.844840] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2583.560483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2583.948707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2585.110238] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2585.175143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2588.641072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.643108] LustreError: Skipped 1 previous similar message [ 2591.840515] Lustre: server umount lustre-OST0000 complete [ 2605.204524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2606.890533] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2608.448461] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2609.380597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.383761] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.383808] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2609.388325] Lustre: Skipped 1 previous similar message [ 2609.823079] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2609.886747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2610.992832] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2611.054939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2612.211797] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a4b077e4000.ost_server_uuid 50 [ 2612.739441] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a4b077e4000.ost_server_uuid in FULL state after 0 sec [ 2612.809151] LustreError: 145992:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.30@tcp) returned error from blocking AST (req@00000000c3885d6f x1845640185440384 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004b7a7583/0x10bfa2ba012111f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.30@tcp remote: 0x9244a691d0e38fed expref: 5 pid: 145991 timeout: 2712 lvb_type: 0 [ 2612.822779] LustreError: lustre-MDT0000: A client on nid 192.168.202.30@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2612.825714] LustreError: 145980:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.30@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004b7a7583/0x10bfa2ba012111f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.30@tcp remote: 0x9244a691d0e38fed expref: 6 pid: 145991 timeout: 0 lvb_type: 0 [ 2614.757176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.759832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2614.764468] Lustre: Skipped 8 previous similar messages [ 2614.770506] Lustre: Skipped 15 previous similar messages [ 2620.079894] Lustre: server umount lustre-OST0000 complete [ 2627.888818] Lustre: server umount lustre-MDT0000 complete [ 2629.675246] LustreError: 145975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140192 with bad export cookie 75428884057820274 [ 2629.680431] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.684883] LustreError: 145975:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2635.944183] Lustre: server umount lustre-MDT0001 complete [ 2640.489410] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2641.935906] Key type lgssc unregistered [ 2642.107593] LNet: 151481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.112366] LNetError: 151481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.122596] LNet: Removed LNI 192.168.202.130@tcp [ 2642.574451] Key type .llcrypt unregistered [ 2642.576294] Key type ._llcrypt unregistered [ 2654.200219] Key type ._llcrypt registered [ 2654.201752] Key type .llcrypt registered [ 2654.242282] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2654.522888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.546052] alg: No test for adler32 (adler32-zlib) [ 2655.413756] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2655.502272] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2657.087181] Key type lgssc registered [ 2657.629251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2661.328496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2661.340580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.570395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.240577] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2668.170582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2668.337411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.002341] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2672.547256] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2674.186034] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2677.419654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2677.579257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.296455] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2682.597874] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2688.499701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2690.799537] Lustre: Failing over lustre-MDT0000 [ 2690.937027] LustreError: 155099:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2690.961481] Lustre: server umount lustre-MDT0000 complete [ 2693.604133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.606142] LustreError: 155009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.611304] Lustre: Skipped 2 previous similar messages [ 2693.621318] LustreError: 155009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2698.720688] LustreError: 152853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.728085] LustreError: 152853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2702.005674] LustreError: 155100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.30@tcp arrived at 1760140264 with bad export cookie 354880177537092677 [ 2702.014604] LustreError: 155100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2703.844382] LustreError: 152853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.853962] LustreError: 152853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2705.207282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.288253] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.440389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.491614] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2707.120720] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2708.157262] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2708.160973] Lustre: lustre-MDT0000: Denying connection for new client be4a4d4a-f2c1-4f62-85d4-72be303a222e (at 192.168.202.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2710.502634] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2715.617867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.618521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2715.621287] Lustre: Skipped 1 previous similar message [ 2720.736826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.738612] Lustre: Skipped 1 previous similar message [ 2720.938799] LustreError: 156031:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2720.940581] LustreError: 156031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2720.998800] Lustre: server umount lustre-OST0000 complete [ 2722.144919] LustreError: 156233:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2722.148765] Lustre: 155599:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2722.151575] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2722.157874] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2722.161589] LustreError: 155599:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a0434f75 x1845640789264640/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2722.166377] LustreError: 155599:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2722.169046] LustreError: 155599:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2722.171809] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2722.256280] LustreError: 156233:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2722.259263] LustreError: 156233:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2722.287046] Lustre: server umount lustre-MDT0000 complete [ 2723.916370] LustreError: 152833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140286 with bad export cookie 354880177537092999 [ 2723.924530] LustreError: 152162:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2723.926175] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.058168] LustreError: 156437:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2730.062332] LustreError: 156437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2730.165943] Lustre: server umount lustre-MDT0001 complete [ 2734.411066] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2735.748502] Key type lgssc unregistered [ 2735.876332] LNet: 156968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.878403] LNetError: 156968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.888293] LNet: Removed LNI 192.168.202.130@tcp [ 2736.166655] Key type .llcrypt unregistered [ 2736.167663] Key type ._llcrypt unregistered [ 2744.891801] Key type ._llcrypt registered [ 2744.892810] Key type .llcrypt registered [ 2744.927054] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2745.198958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.205791] alg: No test for adler32 (adler32-zlib) [ 2746.064506] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2746.166183] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2747.767175] Key type lgssc registered [ 2748.073607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2750.355954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2750.362029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.509275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.954672] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2756.134236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2756.293584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.724876] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2759.653091] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2760.749806] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2762.965449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2763.087866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.198783] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2767.266506] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2777.450915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2781.663869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2781.666086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.671028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2782.688909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.689564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2782.689860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2782.695307] Lustre: Skipped 1 previous similar message [ 2785.388063] LustreError: 160602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2785.438831] Lustre: server umount lustre-OST0000 complete [ 2787.295865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2787.297831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.301150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2792.698951] LustreError: 160804:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2792.701014] LustreError: 160804:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2792.726526] Lustre: server umount lustre-MDT0000 complete [ 2792.927913] LustreError: 159649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.932491] LustreError: 159649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2793.820084] LustreError: 158318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140356 with bad export cookie 16955328357018100301 [ 2793.822844] LustreError: 158318:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2793.823449] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.886446] LustreError: 161005:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2793.888077] LustreError: 161005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2793.957331] Lustre: server umount lustre-MDT0001 complete [ 2797.328267] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2798.259660] Key type lgssc unregistered [ 2798.378360] LNet: 161536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.380278] LNetError: 161536:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.392294] LNet: Removed LNI 192.168.202.130@tcp [ 2798.665132] Key type .llcrypt unregistered [ 2798.666089] Key type ._llcrypt unregistered [ 2811.184112] Key type ._llcrypt registered [ 2811.185617] Key type .llcrypt registered [ 2811.237362] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2811.737124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2811.747532] alg: No test for adler32 (adler32-zlib) [ 2812.624254] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2812.720304] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2814.303182] Key type lgssc registered [ 2814.906411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2818.095243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2818.104227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.274935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.910669] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2824.354995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2824.477300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.676434] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2827.236886] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2828.287351] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2830.811339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2830.919492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.884798] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2834.804632] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2844.269437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2849.761406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.762077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.762559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.764986] Lustre: Skipped 1 previous similar message [ 2853.168133] LustreError: 165462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2853.230109] Lustre: server umount lustre-OST0000 complete [ 2854.880223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.880875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.883227] Lustre: Skipped 1 previous similar message [ 2854.884693] Lustre: Skipped 1 previous similar message [ 2860.000049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.002382] Lustre: Skipped 2 previous similar messages [ 2860.473694] LustreError: 165664:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2860.475404] LustreError: 165664:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2860.498746] Lustre: server umount lustre-MDT0000 complete [ 2861.607927] LustreError: 164815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140424 with bad export cookie 6607815954555895471 [ 2861.610942] LustreError: 164815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2861.611075] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2861.669392] LustreError: 165867:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2861.671018] LustreError: 165867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2861.739967] Lustre: server umount lustre-MDT0001 complete [ 2865.095145] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2865.994934] Key type lgssc unregistered [ 2866.111359] LNet: 166397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.113689] LNetError: 166397:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2866.123271] LNet: Removed LNI 192.168.202.130@tcp [ 2866.387071] Key type .llcrypt unregistered [ 2866.388851] Key type ._llcrypt unregistered [ 2874.208079] Key type ._llcrypt registered [ 2874.209031] Key type .llcrypt registered [ 2874.240324] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2874.534356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2874.568929] alg: No test for adler32 (adler32-zlib) [ 2875.404804] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2875.475158] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2877.047110] Key type lgssc registered [ 2877.337698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2880.413072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2880.418720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.538325] Lustre: *** cfs_fail_loc=135, val=0*** [ 2881.573788] LustreError: 167880:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2881.575622] LustreError: 167880:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.130@tcp: cfg command failed: rc = -2 [ 2881.577933] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2881.577933] [ 2881.580473] LustreError: MGC192.168.202.130@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2881.584025] LustreError: 167839:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2881.586145] LustreError: 167839:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2881.588173] LustreError: 167839:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2881.590495] LustreError: 167839:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2881.644224] Lustre: server umount lustre-MDT0000 complete [ 2881.645515] LustreError: 167839:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2883.614795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.757257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.867680] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2888.528026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2888.725245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.374915] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2892.983565] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2894.595489] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2898.658260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.666973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.670580] Lustre: Skipped 1 previous similar message [ 2901.757825] LustreError: 169595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2901.760130] LustreError: 169595:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2901.787484] Lustre: server umount lustre-MDT0000 complete [ 2902.887251] LustreError: 168231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140465 with bad export cookie 1726217146912592075 [ 2902.889257] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.890228] LustreError: 168231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2902.950717] LustreError: 169798:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2902.952330] LustreError: 169798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2903.023814] Lustre: server umount lustre-MDT0001 complete [ 2906.380270] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2907.288111] Key type lgssc unregistered [ 2907.409355] LNet: 170328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.411447] LNetError: 170328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.418277] LNet: Removed LNI 192.168.202.130@tcp [ 2907.686191] Key type .llcrypt unregistered [ 2907.687172] Key type ._llcrypt unregistered [ 2913.718795] Key type ._llcrypt registered [ 2913.719755] Key type .llcrypt registered [ 2913.753479] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2913.841211] Key type .llcrypt unregistered [ 2913.842038] Key type ._llcrypt unregistered [ 2920.266934] Key type ._llcrypt registered [ 2920.267811] Key type .llcrypt registered [ 2920.299035] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2920.579207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2920.616490] alg: No test for adler32 (adler32-zlib) [ 2921.453374] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 2921.536162] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2923.111097] Key type lgssc registered [ 2923.401423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2925.583119] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2925.588758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2941.023413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2946.143302] LustreError: 172396:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2972.767142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.334446] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2976.031460] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2979.285992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.530292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.569249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2981.280661] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2984.929726] LustreError: 172423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.939519] LustreError: 172423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2985.348574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2985.552700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2987.110882] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2989.454499] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2990.785122] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2992.009293] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3000.287919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.290509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.293874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.801619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.803779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.806931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.362629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.364166] Lustre: Skipped 1 previous similar message [ 3005.680221] LustreError: 175120:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3005.780921] Lustre: server umount lustre-OST0000 complete [ 3008.482259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.483683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.490080] Lustre: Skipped 1 previous similar message [ 3008.496192] Lustre: Skipped 1 previous similar message [ 3013.601261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3013.604870] Lustre: Skipped 1 previous similar message [ 3013.706632] LustreError: 175321:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3013.710573] LustreError: 175321:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3013.741847] Lustre: server umount lustre-MDT0000 complete [ 3015.505120] LustreError: 174723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140578 with bad export cookie 5207906447606958141 [ 3015.511227] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3015.511911] LustreError: 174723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3015.600160] LustreError: 175523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3015.603740] LustreError: 175523:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3015.691944] Lustre: server umount lustre-MDT0001 complete [ 3020.258399] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3021.646267] Key type lgssc unregistered [ 3021.804677] LNet: 176054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3021.809313] LNetError: 176054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3021.819557] LNet: Removed LNI 192.168.202.130@tcp [ 3022.296828] Key type .llcrypt unregistered [ 3022.299052] Key type ._llcrypt unregistered [ 3030.517609] Key type ._llcrypt registered [ 3030.518594] Key type .llcrypt registered [ 3030.548290] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3030.839746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3030.893474] alg: No test for adler32 (adler32-zlib) [ 3031.733286] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3031.802337] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3033.375085] Key type lgssc registered [ 3033.655178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3035.846202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3035.852030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.993342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.079293] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3040.645764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3040.755511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.292824] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3044.218267] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3045.769864] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3049.013448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3049.169086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.608461] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3054.195385] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3061.482645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3066.232764] Lustre: Failing over lustre-MDT0000 [ 3066.329921] LustreError: 179743:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3066.336465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3066.342227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.350665] LustreError: 177424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.357566] LustreError: 177424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3066.372730] Lustre: server umount lustre-MDT0000 complete [ 3066.850565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.853326] LustreError: 177419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.857303] Lustre: Skipped 1 previous similar message [ 3069.745034] LustreError: 177420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.750211] LustreError: 177420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3071.968330] LustreError: 177424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.972130] LustreError: 177424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3077.088842] LustreError: 178137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.092760] LustreError: 178137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3079.363218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.400659] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3079.486545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.513486] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3079.978972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3080.707661] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3082.104408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3084.773425] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3084.780176] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3084.794556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3087.739273] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3092.448319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.452937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.460860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3093.986511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3093.991720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.015648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.019180] Lustre: Skipped 1 previous similar message [ 3097.392770] LustreError: 181066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3097.396525] LustreError: 181066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3097.477258] Lustre: server umount lustre-OST0000 complete [ 3100.128812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.129081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3100.133271] Lustre: Skipped 1 previous similar message [ 3100.137532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3100.139782] Lustre: Skipped 1 previous similar message [ 3105.249041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.252713] Lustre: Skipped 1 previous similar message [ 3105.416251] LustreError: 181271:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3105.420794] LustreError: 181271:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3105.456561] Lustre: server umount lustre-MDT0000 complete [ 3107.168898] LustreError: 177404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140669 with bad export cookie 8225333799236597124 [ 3107.175562] LustreError: 177404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3107.176195] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3107.347624] Lustre: server umount lustre-MDT0001 complete [ 3110.652098] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3111.597937] Key type lgssc unregistered [ 3111.717355] LNet: 182003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3111.719302] LNetError: 182003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3111.730307] LNet: Removed LNI 192.168.202.130@tcp [ 3112.001463] Key type .llcrypt unregistered [ 3112.002511] Key type ._llcrypt unregistered [ 3119.732833] Key type ._llcrypt registered [ 3119.733877] Key type .llcrypt registered [ 3119.774457] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3120.234570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3120.247987] alg: No test for adler32 (adler32-zlib) [ 3121.105725] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3121.189769] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3122.759159] Key type lgssc registered [ 3123.080515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.423222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.429067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.575481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3127.691789] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3130.192877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.300949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3131.402239] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3132.939763] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3133.967245] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3135.883791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3135.985693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.998554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3137.556478] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3139.726907] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3141.470589] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3147.233862] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3154.331333] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3162.591697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3162.593664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.596896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.665086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3165.666246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.666925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.467128] LustreError: 185788:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3167.518712] Lustre: server umount lustre-OST0000 complete [ 3170.784708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3170.785341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3170.787681] Lustre: Skipped 1 previous similar message [ 3170.789264] Lustre: Skipped 1 previous similar message [ 3174.727160] LustreError: 185989:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3174.730463] LustreError: 185989:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3174.766331] Lustre: server umount lustre-MDT0000 complete [ 3175.905629] LustreError: 184882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.914983] LustreError: 184882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3176.468254] LustreError: 184090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140739 with bad export cookie 9246623551914261809 [ 3176.475234] LustreError: 184090:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3176.477443] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3176.555996] LustreError: 186191:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3176.559725] LustreError: 186191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3176.687451] Lustre: server umount lustre-MDT0001 complete [ 3180.654945] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3182.104837] Key type lgssc unregistered [ 3182.252598] LNet: 186722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3182.256875] LNetError: 186722:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3182.267666] LNet: Removed LNI 192.168.202.130@tcp [ 3182.690200] Key type .llcrypt unregistered [ 3182.692261] Key type ._llcrypt unregistered [ 3192.008967] Key type ._llcrypt registered [ 3192.010150] Key type .llcrypt registered [ 3192.053424] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3192.505336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3192.515628] alg: No test for adler32 (adler32-zlib) [ 3193.410771] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3193.502496] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3195.079167] Key type lgssc registered [ 3195.674941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3199.359449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3199.365263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.545670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.308840] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3206.003078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3206.205143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3207.812554] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3209.545481] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3210.995691] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3213.522526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.659160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.653764] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3217.932866] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3226.978555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3229.697591] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3239.555582] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3249.735697] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3257.144740] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3267.039749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.041693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.045321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.066409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.066739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.066777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.185396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.186953] Lustre: Skipped 2 previous similar messages [ 3278.305735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.307286] Lustre: Skipped 1 previous similar message [ 3280.351100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3280.366980] LustreError: 192479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3280.419066] Lustre: server umount lustre-OST0000 complete [ 3282.911679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3282.913666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.916898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.622331] LustreError: 192683:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3287.625375] LustreError: 192683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3287.653995] Lustre: server umount lustre-MDT0000 complete [ 3288.543930] LustreError: 188092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.547693] LustreError: 188092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3288.808698] LustreError: 188072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140851 with bad export cookie 7994549035070788679 [ 3288.814606] LustreError: 188072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3288.814988] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.895386] LustreError: 192884:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3288.897873] LustreError: 192884:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3288.971548] Lustre: server umount lustre-MDT0001 complete [ 3291.941950] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3292.841506] Key type lgssc unregistered [ 3292.959344] LNet: 193415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3292.963474] LNetError: 193415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3292.974283] LNet: Removed LNI 192.168.202.130@tcp [ 3293.237462] Key type .llcrypt unregistered [ 3293.238356] Key type ._llcrypt unregistered [ 3302.028909] Key type ._llcrypt registered [ 3302.029951] Key type .llcrypt registered [ 3302.065344] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3302.374290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3302.407077] alg: No test for adler32 (adler32-zlib) [ 3303.242537] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3303.314151] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3304.887087] Key type lgssc registered [ 3305.182528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.446742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.452786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.601136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.785584] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3312.488351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3312.597909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3313.724071] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3315.280557] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3316.277331] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3318.209503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3318.314172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.334570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3319.968684] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3321.500874] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3328.485405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.485624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.488912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.492295] Lustre: Skipped 1 previous similar message [ 3328.495295] Lustre: Skipped 1 previous similar message [ 3332.338671] LustreError: 197194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3332.431448] Lustre: server umount lustre-OST0000 complete [ 3338.207791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3338.209826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.213280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3339.900243] LustreError: 197398:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3339.901905] LustreError: 197398:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3339.929716] Lustre: server umount lustre-MDT0000 complete [ 3341.055108] LustreError: 194778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140903 with bad export cookie 7255911527657691890 [ 3341.056929] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.058030] LustreError: 194778:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3341.127707] LustreError: 197600:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3341.129396] LustreError: 197600:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3341.220709] Lustre: server umount lustre-MDT0001 complete [ 3344.405558] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3345.440541] Key type lgssc unregistered [ 3345.578578] LNet: 198130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3345.580591] LNetError: 198130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3345.589469] LNet: Removed LNI 192.168.202.130@tcp [ 3345.875040] Key type .llcrypt unregistered [ 3345.875965] Key type ._llcrypt unregistered [ 3352.280158] Key type ._llcrypt registered [ 3352.281161] Key type .llcrypt registered [ 3352.316125] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3352.870252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.879752] alg: No test for adler32 (adler32-zlib) [ 3353.725660] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3353.805219] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3355.383122] Key type lgssc registered [ 3355.699545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3357.899527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3357.907953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.060781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.133291] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3362.569394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3362.684542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.785080] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3365.287647] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3366.280613] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3368.210542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3368.328331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.383193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3369.912611] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3371.430798] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3377.633047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.634152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.635621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.639893] Lustre: Skipped 1 previous similar message [ 3377.642592] Lustre: Skipped 1 previous similar message [ 3381.420161] LustreError: 201748:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3381.473139] Lustre: server umount lustre-OST0000 complete [ 3382.752298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.752892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.755777] Lustre: Skipped 1 previous similar message [ 3387.871961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.873566] Lustre: Skipped 2 previous similar messages [ 3388.662835] LustreError: 201950:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3388.664758] LustreError: 201950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3388.685377] Lustre: server umount lustre-MDT0000 complete [ 3389.699930] LustreError: 201952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760140952 with bad export cookie 4914044329288668882 [ 3389.702440] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.702796] LustreError: 201952:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3389.766315] LustreError: 202153:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3389.767850] LustreError: 202153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3389.839892] Lustre: server umount lustre-MDT0001 complete [ 3393.071302] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3393.963167] Key type lgssc unregistered [ 3394.087330] LNet: 202684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3394.089230] LNetError: 202684:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3394.097280] LNet: Removed LNI 192.168.202.130@tcp [ 3394.351324] Key type .llcrypt unregistered [ 3394.352262] Key type ._llcrypt unregistered [ 3400.733889] Key type ._llcrypt registered [ 3400.734803] Key type .llcrypt registered [ 3400.765259] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3401.042597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3401.074077] alg: No test for adler32 (adler32-zlib) [ 3401.908985] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3401.981845] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3403.551134] Key type lgssc registered [ 3403.826587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3406.063240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3406.069267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.216793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.297956] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3410.767951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3410.881863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3411.956068] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3413.449524] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3414.460529] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3416.396987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3416.507398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.574288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3418.080436] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3419.668286] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3422.810771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3422.832774] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3422.894209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3424.482948] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3426.095319] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3427.643228] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3438.157277] Lustre: Permanently deactivating lustre-OST0001 [ 3438.162411] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3446.772463] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3448.753057] Lustre: Permanently reactivating lustre-OST0001 [ 3448.758195] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3448.759828] Lustre: Skipped 2 previous similar messages [ 3452.422374] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.428194] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3452.430892] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3452.434878] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3452.446544] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3462.624558] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3462.624609] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.626847] LustreError: Skipped 1 previous similar message [ 3462.629705] Lustre: Skipped 1 previous similar message [ 3462.630377] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3462.634268] Lustre: Skipped 1 previous similar message [ 3464.674824] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3464.676406] Lustre: Skipped 1 previous similar message [ 3466.089366] LustreError: 207368:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3466.104794] Lustre: server umount lustre-OST0001 complete [ 3468.767704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3468.769741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.772642] Lustre: Skipped 1 previous similar message [ 3468.774081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3469.795334] LustreError: 207370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.799027] LustreError: 207370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3473.261099] LustreError: 207570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3473.262765] LustreError: 207570:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3473.316081] Lustre: server umount lustre-OST0000 complete [ 3474.912511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.912925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.915468] Lustre: Skipped 2 previous similar messages [ 3474.916881] Lustre: Skipped 2 previous similar messages [ 3480.033190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.037306] Lustre: Skipped 1 previous similar message [ 3480.526080] LustreError: 207771:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3480.529783] LustreError: 207771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3480.564672] Lustre: server umount lustre-MDT0000 complete [ 3482.145221] LustreError: 203887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141044 with bad export cookie 1757139028705370837 [ 3482.151911] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.152067] LustreError: 203887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3482.324829] Lustre: server umount lustre-MDT0001 complete [ 3487.268315] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3488.737942] Key type lgssc unregistered [ 3488.906855] LNet: 208505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.911479] LNetError: 208505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.922687] LNet: Removed LNI 192.168.202.130@tcp [ 3489.413808] Key type .llcrypt unregistered [ 3489.415182] Key type ._llcrypt unregistered [ 3505.006183] Key type ._llcrypt registered [ 3505.008136] Key type .llcrypt registered [ 3505.075348] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 3511.556893] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3511.832872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3511.873640] alg: No test for adler32 (adler32-zlib) [ 3512.722912] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3512.801056] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3514.375092] Key type lgssc registered [ 3514.767758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3517.491734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3519.496198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3522.062433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3525.111973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3528.376652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3528.401330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3528.413432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3529.548453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3529.568365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3529.623403] Lustre: lustre-MDT0000: new disk, initializing [ 3529.658584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.669212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3531.454744] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3536.761924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3536.802084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3536.852193] Lustre: 212423:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3536.894750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3536.897781] Lustre: Skipped 1 previous similar message [ 3536.943491] Lustre: lustre-MDT0001: new disk, initializing [ 3536.996438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3537.017989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3537.024343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3538.303861] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3541.142917] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3542.169578] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3544.227217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.254840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3544.355805] Lustre: lustre-OST0000: new disk, initializing [ 3544.358195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3544.388159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.956600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3544.959101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3545.021595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3546.079179] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3549.030682] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3550.216186] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3550.286501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3551.417119] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3551.491232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3555.300242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.301052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3555.302315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3555.303130] Lustre: Skipped 1 previous similar message [ 3555.304992] LustreError: Skipped 1 previous similar message [ 3558.123141] LustreError: 215359:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3558.184223] Lustre: server umount lustre-OST0000 complete [ 3560.418924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.420432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.425835] Lustre: Skipped 1 previous similar message [ 3560.429239] Lustre: Skipped 1 previous similar message [ 3565.537405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.539020] Lustre: Skipped 1 previous similar message [ 3570.657292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3570.658826] Lustre: Skipped 1 previous similar message [ 3574.239082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3574.270474] LustreError: 215560:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3574.273524] LustreError: 215560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3574.302707] Lustre: server umount lustre-MDT0000 complete [ 3575.462489] LustreError: 212415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141138 with bad export cookie 13463287481273086663 [ 3575.465374] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.466173] LustreError: 212415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3575.532876] LustreError: 215762:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 3575.534448] LustreError: 215762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3575.618708] Lustre: server umount lustre-MDT0001 complete [ 3578.038118] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 20:05:40 (1760141140) [ 3580.023516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.169374] LustreError: 216432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.174507] LustreError: 216432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3580.191978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.333057] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3583.926699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3585.191161] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3586.841076] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3587.869979] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3589.930524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3590.043347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.045134] Lustre: Skipped 1 previous similar message [ 3591.723734] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3593.270253] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3594.756392] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3601.222814] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3607.694522] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3617.296285] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3625.797463] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3642.759709] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3642.761663] Lustre: Skipped 1 previous similar message [ 3674.856128] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3674.858097] Lustre: Skipped 3 previous similar messages [ 3739.206544] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3739.208322] Lustre: Skipped 7 previous similar messages [ 3763.171962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.173416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.174590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.176706] Lustre: Skipped 1 previous similar message [ 3767.340990] LustreError: 219919:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3767.342698] LustreError: 219919:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3767.403251] Lustre: server umount lustre-OST0000 complete [ 3773.409926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.410111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3773.410808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.410816] Lustre: Skipped 1 previous similar message [ 3773.412775] Lustre: Skipped 1 previous similar message [ 3774.653732] LustreError: 220124:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3774.655234] LustreError: 220124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3774.681883] Lustre: server umount lustre-MDT0000 complete [ 3775.771516] LustreError: 218626:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141338 with bad export cookie 13463287481273087475 [ 3775.776768] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3775.926484] Lustre: server umount lustre-MDT0001 complete [ 3779.115902] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3780.135944] Key type lgssc unregistered [ 3780.272299] LNet: 220857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3780.274370] LNetError: 220857:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3780.281516] LNet: Removed LNI 192.168.202.130@tcp [ 3780.586972] Key type .llcrypt unregistered [ 3780.587968] Key type ._llcrypt unregistered [ 3788.482677] Key type ._llcrypt registered [ 3788.483556] Key type .llcrypt registered [ 3788.521698] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3788.912847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3788.961465] alg: No test for adler32 (adler32-zlib) [ 3789.802964] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3789.874634] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3791.447092] Key type lgssc registered [ 3791.738481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3793.988394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3793.994459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.147212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.235030] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3798.718076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3798.833837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3799.945949] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3801.472560] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3802.476807] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3804.470546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3804.587947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.280291] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3807.825376] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3818.984607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.992368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3818.992871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3819.000600] Lustre: Skipped 1 previous similar message [ 3820.068051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.216184] LustreError: 224631:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3823.320603] Lustre: server umount lustre-OST0000 complete [ 3825.121082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.122050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.127330] Lustre: Skipped 2 previous similar messages [ 3825.129975] Lustre: Skipped 1 previous similar message [ 3831.244554] LustreError: 224837:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3831.248331] LustreError: 224837:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3831.264554] LustreError: 222227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.270443] LustreError: 222227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3831.287282] Lustre: server umount lustre-MDT0000 complete [ 3833.054269] LustreError: 224469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1760141395 with bad export cookie 2190361247025177218 [ 3833.059465] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.061108] LustreError: 224469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3833.162108] LustreError: 225039:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3833.165538] LustreError: 225039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3833.264281] Lustre: server umount lustre-MDT0001 complete [ 3838.062510] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3839.415878] Key type lgssc unregistered [ 3839.575641] LNet: 225579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3839.580516] LNetError: 225579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3839.592609] LNet: Removed LNI 192.168.202.130@tcp [ 3840.057244] Key type .llcrypt unregistered [ 3840.059272] Key type ._llcrypt unregistered [ 3846.464405] Key type ._llcrypt registered [ 3846.465310] Key type .llcrypt registered [ 3846.496725] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3846.592222] Key type .llcrypt unregistered [ 3846.593130] Key type ._llcrypt unregistered [ 3854.685953] Key type ._llcrypt registered [ 3854.687665] Key type .llcrypt registered [ 3854.737802] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 3862.481486] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3862.822398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3862.876588] alg: No test for adler32 (adler32-zlib) [ 3863.710201] Lustre: Lustre: Build Version: 2.16.58_105_gc3ac3ca [ 3863.786419] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3865.359145] Key type lgssc registered [ 3865.887282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3869.074073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3871.520200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3873.542332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3875.729206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3876.318237] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3749 sec ========== 20:10:38 (1760141438) [ 3876.874853] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:10:39 (1760141439) === [ 3877.952686] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:10:40 (1760141440) === [ 3889.743911] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3890.662979] Key type lgssc unregistered [ 3890.774375] LNet: 231657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3890.776223] LNetError: 231657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3890.788290] LNet: Removed LNI 192.168.202.130@tcp [ 3891.045309] Key type .llcrypt unregistered [ 3891.046292] Key type ._llcrypt unregistered