[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 502225249 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2624MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003510] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008535] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011126] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.022295] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027633] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030097] *** VALIDATE tmpfs *** [ 0.032404] *** VALIDATE proc *** [ 0.033206] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037094] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039026] Spectre V2 : User space: Vulnerable [ 0.040000] Speculative Store Bypass: Vulnerable [ 0.042330] debug: unmapping init [mem 0xffffffffa6259000-0xffffffffa6260fff] [ 0.044793] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048007] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052006] ... event mask: 000000070000000f [ 0.053256] rcu: Hierarchical SRCU implementation. [ 0.056069] smp: Bringing up secondary CPUs ... [ 0.057517] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.061223] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142800] node 0 deferred pages initialised in 76ms [ 0.146010] devtmpfs: initialized [ 0.147215] x86/mm: Memory block size: 128MB [ 0.150155] gcov: version magic: 0x41383552 [ 0.152086] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154234] pinctrl core: initialized pinctrl subsystem [ 0.155190] [ 0.155704] ************************************************************* [ 0.156014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ** ** [ 0.158012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159012] ** ** [ 0.160013] ** This means that this kernel is built to expose internal ** [ 0.161010] ** IOMMU data structures, which may compromise security on ** [ 0.162010] ** your system. ** [ 0.163012] ** ** [ 0.164015] ** If you see this message and you are not debugging the ** [ 0.165012] ** kernel, report this immediately to your vendor! ** [ 0.166013] ** ** [ 0.167011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168011] ************************************************************* [ 0.169814] NET: Registered protocol family 16 [ 0.170582] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180133] cpuidle: using governor menu [ 0.181837] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184642] PCI: Using configuration type 1 for base access [ 0.187114] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199161] cryptd: max_cpu_qlen set to 1000 [ 0.202242] ACPI: Added _OSI(Module Device) [ 0.204015] ACPI: Added _OSI(Processor Device) [ 0.205011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211516] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218541] ACPI: Interpreter enabled [ 0.220058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221011] ACPI: Using IOAPIC for interrupt routing [ 0.223089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234816] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248142] acpiphp: Slot [2] registered [ 0.249090] acpiphp: Slot [3] registered [ 0.250062] acpiphp: Slot [4] registered [ 0.251081] acpiphp: Slot [5] registered [ 0.252109] acpiphp: Slot [6] registered [ 0.254110] acpiphp: Slot [7] registered [ 0.255092] acpiphp: Slot [8] registered [ 0.256073] acpiphp: Slot [9] registered [ 0.258089] acpiphp: Slot [10] registered [ 0.259083] acpiphp: Slot [11] registered [ 0.261079] acpiphp: Slot [12] registered [ 0.262086] acpiphp: Slot [13] registered [ 0.263077] acpiphp: Slot [14] registered [ 0.264077] acpiphp: Slot [15] registered [ 0.266058] acpiphp: Slot [16] registered [ 0.267087] acpiphp: Slot [17] registered [ 0.268083] acpiphp: Slot [18] registered [ 0.270069] acpiphp: Slot [19] registered [ 0.271098] acpiphp: Slot [20] registered [ 0.273127] acpiphp: Slot [21] registered [ 0.274091] acpiphp: Slot [22] registered [ 0.276101] acpiphp: Slot [23] registered [ 0.278125] acpiphp: Slot [24] registered [ 0.279080] acpiphp: Slot [25] registered [ 0.281157] acpiphp: Slot [26] registered [ 0.282097] acpiphp: Slot [27] registered [ 0.284161] acpiphp: Slot [28] registered [ 0.285086] acpiphp: Slot [29] registered [ 0.287088] acpiphp: Slot [30] registered [ 0.288118] acpiphp: Slot [31] registered [ 0.290053] PCI host bridge to bus 0000:00 [ 0.291018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301541] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.303023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309342] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.313575] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.325476] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.331069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.333016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.336017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.338019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.341452] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343771] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.346038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348619] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.365016] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.370814] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375666] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408016] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.417561] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.456098] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.462012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.487016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.498831] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.504012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.508012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.524017] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.531548] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.538013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.542010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.553013] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.562199] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.568015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.575018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.588016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.598714] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.601398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.604412] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.606353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.609217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.613706] iommu: Default domain type: Passthrough [ 0.617323] SCSI subsystem initialized [ 0.618128] ACPI: bus type USB registered [ 0.620098] usbcore: registered new interface driver usbfs [ 0.622077] usbcore: registered new interface driver hub [ 0.624077] usbcore: registered new device driver usb [ 0.626182] pps_core: LinuxPPS API ver. 1 registered [ 0.628010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.631079] PTP clock support registered [ 0.633115] EDAC MC: Ver: 3.0.0 [ 0.634115] PCI: Using ACPI for IRQ routing [ 0.636071] NetLabel: Initializing [ 0.637009] NetLabel: domain hash size = 128 [ 0.639010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.641076] NetLabel: unlabeled traffic allowed by default [ 0.644043] vgaarb: loaded [ 0.645251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.647015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.656430] clocksource: Switched to clocksource kvm-clock [ 0.761886] VFS: Disk quotas dquot_6.6.0 [ 0.763436] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.765807] *** VALIDATE ramfs *** [ 0.767060] *** VALIDATE hugetlbfs *** [ 0.768559] pnp: PnP ACPI init [ 0.771335] pnp: PnP ACPI: found 6 devices [ 0.789884] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.792903] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.794858] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.796691] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.798873] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.800991] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.803398] NET: Registered protocol family 2 [ 0.805641] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.810078] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.813611] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.819493] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.824058] TCP: Hash tables configured (established 65536 bind 65536) [ 0.827459] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.833192] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.836054] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.838473] NET: Registered protocol family 1 [ 0.840506] RPC: Registered named UNIX socket transport module. [ 0.843274] RPC: Registered udp transport module. [ 0.844502] RPC: Registered tcp transport module. [ 0.845729] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.848212] NET: Registered protocol family 44 [ 0.849602] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.852292] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.854923] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.857420] PCI: CLS 0 bytes, default 64 [ 0.859497] Unpacking initramfs... [ 2.323566] debug: unmapping init [mem 0xffff96897cc54000-0xffff96897ffbffff] [ 2.327949] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.330592] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.333757] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.836121] Initialise system trusted keyrings [ 2.837752] Key type blacklist registered [ 2.841920] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.853867] zbud: loaded [ 2.858678] *** VALIDATE nfs *** [ 2.860102] *** VALIDATE nfs4 *** [ 2.862200] pstore: using deflate compression [ 2.866938] Platform Keyring initialized [ 2.971507] NET: Registered protocol family 38 [ 2.973257] Key type asymmetric registered [ 2.974641] Asymmetric key parser 'x509' registered [ 2.976397] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.979248] io scheduler mq-deadline registered [ 2.980824] io scheduler kyber registered [ 2.982294] io scheduler bfq registered [ 2.984527] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.987223] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.991069] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.994126] ACPI: Power Button [PWRF] [ 3.090126] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.191757] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.372613] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.464903] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.646735] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.678478] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.712575] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.717771] Non-volatile memory driver v1.3 [ 3.719303] Linux agpgart interface v0.103 [ 3.750521] virtio_blk virtio1: [vda] 133896 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.753500] vda: detected capacity change from 0 to 68554752 [ 3.766676] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.769604] vdb: detected capacity change from 0 to 1073741824 [ 3.784524] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.787547] vdc: detected capacity change from 0 to 2621440000 [ 3.802516] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.805595] vdd: detected capacity change from 0 to 2621440000 [ 3.819569] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.822315] vde: detected capacity change from 0 to 4294967296 [ 3.835489] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.838408] vdf: detected capacity change from 0 to 4294967296 [ 3.844302] libphy: Fixed MDIO Bus: probed [ 3.851284] usbcore: registered new interface driver usbserial_generic [ 3.853660] usbserial: USB Serial support registered for generic [ 3.859506] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.863558] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.865858] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.869799] mousedev: PS/2 mouse device common for all mice [ 3.872736] rtc_cmos 00:05: RTC can wake from S4 [ 3.875677] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.879368] rtc_cmos 00:05: registered as rtc0 [ 3.880937] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.885641] intel_pstate: CPU model not supported [ 3.890609] hid: raw HID events driver (C) Jiri Kosina [ 3.894527] usbcore: registered new interface driver usbhid [ 3.896343] usbhid: USB HID core driver [ 3.897641] drop_monitor: Initializing network drop monitor service [ 3.899517] Initializing XFRM netlink socket [ 3.900898] NET: Registered protocol family 10 [ 3.903430] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.912681] Segment Routing with IPv6 [ 3.912715] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.914075] NET: Registered protocol family 17 [ 3.919493] mpls_gso: MPLS GSO support [ 3.925156] RAS: Correctable Errors collector initialized. [ 3.927044] AVX version of gcm_enc/dec engaged. [ 3.929067] AES CTR mode by8 optimization enabled [ 4.005525] sched_clock: Marking stable (4005501769, 0)->(4942867139, -937365370) [ 4.010092] registered taskstats version 1 [ 4.012190] Loading compiled-in X.509 certificates [ 4.014325] zswap: loaded using pool lzo/zbud [ 4.038199] Key type big_key registered [ 4.051595] Key type encrypted registered [ 4.053571] ima: No TPM chip found, activating TPM-bypass! [ 4.056029] ima: Allocated hash algorithm: sha1 [ 4.058058] ima: No architecture policies found [ 4.060010] evm: Initialising EVM extended attributes: [ 4.062183] evm: security.selinux [ 4.063510] evm: security.ima [ 4.064731] evm: security.capability [ 4.066698] evm: HMAC attrs: 0x1 [ 4.069821] rtc_cmos 00:05: setting system clock to 2025-10-23 18:43:45 UTC (1761245025) [ 4.078230] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 4.082643] debug: unmapping init [mem 0xffffffffa5f82000-0xffffffffa6258fff] [ 4.091813] Write protecting the kernel read-only data: 28672k [ 4.096794] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 4.100811] debug: unmapping init [mem 0xffffffffa4f14000-0xffffffffa4ffffff] [ 4.136698] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.147277] systemd[1]: Detected virtualization kvm. [ 4.149411] systemd[1]: Detected architecture x86-64. [ 4.151573] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.176125] systemd[1]: No hostname configured. [ 4.177369] systemd[1]: Set hostname to . [ 4.179379] random: systemd: uninitialized urandom read (16 bytes read) [ 4.181460] systemd[1]: Initializing machine ID from random generator. [ 4.227201] random: ln: uninitialized urandom read (6 bytes read) [ 4.310737] random: systemd: uninitialized urandom read (16 bytes read) [ 4.312942] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.318570] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.322290] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.886943] device-mapper: uevent: version 1.0.3 [ 4.889491] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.722777] virtio_net virtio0 ens2: renamed from eth0 [ 5.723695] random: fast init done [ 5.797070] scsi host0: ata_piix [ 5.806177] scsi host1: ata_piix [ 5.807651] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.810238] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.592867] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.573653] random: crng init done [ 10.575190] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.061627] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.230442] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.519437] SELinux: Disabled at runtime. [ 12.580612] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.589709] systemd[1]: Detected virtualization kvm. [ 12.591525] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.193835] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.198654] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.204840] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.211375] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.219374] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.229974] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.253895] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 13.427320] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.765077] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.117263] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.204223] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.349839] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.364454] EDAC sbridge: Ver: 1.1.2 [ 18.130270] Key type dns_resolver registered [ 18.439543] NFS: Registering the id_resolver key type [ 18.441580] Key type id_resolver registered [ 18.443018] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg360-server login: [ 50.054057] hrtimer: interrupt took 3230187 ns [ 56.730420] libcfs: loading out-of-tree module taints kernel. [ 56.773995] Key type ._llcrypt registered [ 56.776904] Key type .llcrypt registered [ 56.874301] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 72.052158] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 73.334067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.349349] alg: No test for adler32 (adler32-zlib) [ 74.732615] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 75.586459] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 77.456223] Key type lgssc registered [ 78.952478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.761741] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.597523] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.864679] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.829836] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.889425] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.624149] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 144.336785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.391234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 144.413562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.632419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 145.662286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 145.739788] Lustre: lustre-MDT0000: new disk, initializing [ 145.824250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.877964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 149.579292] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.042838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.108396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 163.163715] Lustre: 6512:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.185073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.187454] Lustre: Skipped 1 previous similar message [ 163.252874] Lustre: lustre-MDT0001: new disk, initializing [ 163.330391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.358286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.370628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 167.918801] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.885589] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.140830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 181.311815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 181.546608] Lustre: lustre-OST0000: new disk, initializing [ 181.552945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.601251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.531875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.549343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.775044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 189.079094] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.360456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.428107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.539592] Lustre: lustre-OST0001: new disk, initializing [ 205.542622] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.627111] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.989496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.999534] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.075731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.227710] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.720547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.886316] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.729441] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing check_logdir /tmp/testlogs/ [ 242.225777] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing yml_node [ 245.903665] Lustre: DEBUG MARKER: Client: 2.16.59.45 [ 248.415218] Lustre: DEBUG MARKER: MDS: 2.16.59.45 [ 251.036903] Lustre: DEBUG MARKER: OSS: 2.16.59.45 [ 252.645492] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 23 14:47:52 EDT 2025 [ 270.823985] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 272.424594] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.712919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.714186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 287.730876] Lustre: Skipped 3 previous similar messages [ 287.740105] Lustre: Skipped 3 previous similar messages [ 292.835603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 292.844478] Lustre: Skipped 3 previous similar messages [ 293.790396] LustreError: 12225:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 293.966773] Lustre: server umount lustre-MDT0000 complete [ 297.681631] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245319 with bad export cookie 4964666240624679266 [ 297.691062] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.713275] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.952985] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.956859] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 297.983716] Lustre: Skipped 1 previous similar message [ 303.074205] LustreError: 6518:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.077479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.109541] LustreError: 6518:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 303.135110] Lustre: Skipped 2 previous similar messages [ 303.968917] LustreError: 12426:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 303.981871] LustreError: 12426:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 304.272105] Lustre: server umount lustre-MDT0001 complete [ 308.171220] LustreError: 12628:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 308.175250] LustreError: 12628:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 308.280793] Lustre: server umount lustre-OST0000 complete [ 312.751855] LustreError: 12828:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 312.760596] LustreError: 12828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 313.123223] Lustre: server umount lustre-OST0001 complete [ 319.732417] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 329.096581] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 339.181574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.610167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 351.566650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 357.260332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.232908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.308348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.630505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.663189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.800714] Lustre: lustre-MDT0000: new disk, initializing [ 364.926695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.965343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 369.411757] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 380.698158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 380.798169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 380.989288] Lustre: 15012:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 381.111345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 381.124823] Lustre: Skipped 1 previous similar message [ 381.250916] Lustre: lustre-MDT0001: new disk, initializing [ 381.416959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 381.474386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 381.487591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 381.595145] Lustre: 15811:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 381.609693] Lustre: 15811:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 381.612848] Lustre: 15811:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 381.616159] Lustre: 15811:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 381.619714] Lustre: 15811:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 381.623140] Lustre: 15811:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 385.363331] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 392.152436] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 395.990303] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 402.631670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 402.734914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 402.981425] Lustre: lustre-OST0000: new disk, initializing [ 402.992126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 403.094074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.561710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.574977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 404.719429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 408.597074] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 416.297942] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 421.113750] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 421.472369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 426.222034] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 426.501717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.563213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.570697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.581779] Lustre: Skipped 1 previous similar message [ 430.599217] Lustre: Skipped 1 previous similar message [ 434.790507] LustreError: 17953:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 434.800455] LustreError: 17953:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 435.107589] Lustre: server umount lustre-OST0000 complete [ 440.804734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.811831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.814380] Lustre: Skipped 1 previous similar message [ 445.866736] LustreError: 18155:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 445.878467] LustreError: 18155:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 445.922731] LustreError: 15025:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.957670] LustreError: 15025:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 446.047383] Lustre: server umount lustre-MDT0000 complete [ 449.904159] LustreError: 15005:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245471 with bad export cookie 4964666240624680841 [ 449.917489] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.248147] Lustre: server umount lustre-MDT0001 complete [ 453.595328] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:51:13 (1761245473) [ 460.257595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.782399] LustreError: 18884:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.908583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.507661] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 465.889335] LustreError: 18884:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.010134] LustreError: 18884:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.048112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 478.219596] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 484.229410] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 488.279886] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.603941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 501.424517] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 506.183855] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 511.903094] LustreError: 21130:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 511.906243] LustreError: 21130:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 512.118765] Lustre: server umount lustre-OST0000 complete [ 516.579722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.586873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.594488] Lustre: Skipped 2 previous similar messages [ 516.605305] Lustre: Skipped 1 previous similar message [ 525.887605] Lustre: server umount lustre-MDT0000 complete [ 526.819972] LustreError: 18884:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.838964] LustreError: 18884:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 529.169986] LustreError: 18863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245550 with bad export cookie 4964666240624681765 [ 529.188368] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.190748] LustreError: 18863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 539.262756] Lustre: server umount lustre-MDT0001 complete [ 551.822607] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 554.918675] Key type lgssc unregistered [ 555.195707] LNet: 22069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.205559] LNetError: 22069:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.228599] LNet: Removed LNI 192.168.203.160@tcp [ 556.203118] Key type .llcrypt unregistered [ 556.205824] Key type ._llcrypt unregistered [ 576.744680] Key type ._llcrypt registered [ 576.747340] Key type .llcrypt registered [ 576.836621] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 577.915928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 577.955437] alg: No test for adler32 (adler32-zlib) [ 579.085262] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 579.371580] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 581.048257] Key type lgssc registered [ 582.146539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 589.913175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 589.933293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.467036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 595.371943] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 601.092513] LustreError: 23441:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.119936] LustreError: 23441:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 604.030581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 604.457151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 608.416306] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 613.573214] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 617.143599] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 623.910510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 624.226084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.651882] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 634.898591] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 636.550300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 650.209761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 650.213436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.219786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 651.338165] LustreError: 26134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 651.562531] Lustre: server umount lustre-OST0000 complete [ 655.846723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 655.857942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.884080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.377214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.380186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.497142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 662.507857] Lustre: Skipped 2 previous similar messages [ 665.588396] LustreError: 26336:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 665.591079] LustreError: 26336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 665.690142] Lustre: server umount lustre-MDT0000 complete [ 666.600505] LustreError: 23441:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.616137] LustreError: 23441:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 670.731746] LustreError: 23423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245692 with bad export cookie 10143962716416780736 [ 670.735881] LustreError: 23423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 670.736876] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.971189] LustreError: 26538:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 671.015138] LustreError: 26538:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 671.298361] Lustre: server umount lustre-MDT0001 complete [ 684.147313] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 687.227848] Key type lgssc unregistered [ 687.568163] LNet: 27070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.586069] LNetError: 27070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.614252] LNet: Removed LNI 192.168.203.160@tcp [ 689.368926] Key type .llcrypt unregistered [ 689.375374] Key type ._llcrypt unregistered [ 710.669372] Key type ._llcrypt registered [ 710.672378] Key type .llcrypt registered [ 710.784710] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 711.979517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.080927] alg: No test for adler32 (adler32-zlib) [ 713.057302] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 713.350839] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 715.016164] Key type lgssc registered [ 716.266589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 724.455506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 724.475045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.928464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 730.229403] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 736.230793] LustreError: 28464:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.266767] LustreError: 28464:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 739.164680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 739.510627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.954919] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 749.736133] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 752.910550] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 765.048684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 765.433711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.605994] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 771.569457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 775.696164] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 792.034252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.046819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.061590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 794.083895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 794.093270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 794.103813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.207095] LustreError: 31080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 797.443624] Lustre: server umount lustre-OST0000 complete [ 801.249157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 801.255168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.278259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 801.280806] Lustre: Skipped 1 previous similar message [ 804.321332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 804.322825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 804.354862] Lustre: Skipped 1 previous similar message [ 806.696800] LustreError: 31282:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 806.699993] LustreError: 31282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 806.753955] Lustre: server umount lustre-MDT0000 complete [ 809.308780] LustreError: 31081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245830 with bad export cookie 644297318697832452 [ 809.310609] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.321289] LustreError: 31081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 809.413199] LustreError: 31484:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 809.423382] LustreError: 31484:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 809.602876] Lustre: server umount lustre-MDT0001 complete [ 818.146513] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 820.060220] Key type lgssc unregistered [ 820.275941] LNet: 32016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.284517] LNetError: 32016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.300367] LNet: Removed LNI 192.168.203.160@tcp [ 820.895443] Key type .llcrypt unregistered [ 820.902378] Key type ._llcrypt unregistered [ 840.882583] Key type ._llcrypt registered [ 840.883965] Key type .llcrypt registered [ 841.028564] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 841.909868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.998710] alg: No test for adler32 (adler32-zlib) [ 842.974378] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 843.254570] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 844.976178] Key type lgssc registered [ 846.252445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 855.393225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 855.415611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.253724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.393917] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 866.815022] LustreError: 33386:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.830035] LustreError: 33386:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 870.458181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 870.877687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.458685] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 879.425073] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 882.431932] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 887.922969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 888.161478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.851211] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 896.785367] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 902.523502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 911.847230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 911.850653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.860817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.865632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 912.886575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.890043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.914751] Lustre: Skipped 1 previous similar message [ 914.174099] LustreError: 35638:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 914.326413] Lustre: server umount lustre-OST0000 complete [ 917.986163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.987980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 917.995411] Lustre: Skipped 1 previous similar message [ 918.018834] Lustre: Skipped 1 previous similar message [ 923.105695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.109604] Lustre: Skipped 1 previous similar message [ 923.609857] LustreError: 35840:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 923.615716] LustreError: 35840:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 923.686267] Lustre: server umount lustre-MDT0000 complete [ 926.188737] LustreError: 33366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245947 with bad export cookie 4404126049785381469 [ 926.191061] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 926.197570] LustreError: 33366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 926.306355] LustreError: 36041:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 926.309192] LustreError: 36041:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 926.488377] Lustre: server umount lustre-MDT0001 complete [ 936.905666] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 938.925493] Key type lgssc unregistered [ 939.199803] LNet: 36574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.210978] LNetError: 36574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.229943] LNet: Removed LNI 192.168.203.160@tcp [ 939.914901] Key type .llcrypt unregistered [ 939.916949] Key type ._llcrypt unregistered [ 957.724977] Key type ._llcrypt registered [ 957.726535] Key type .llcrypt registered [ 957.806307] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 958.579425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.598321] alg: No test for adler32 (adler32-zlib) [ 959.568879] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 959.713942] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 961.345641] Key type lgssc registered [ 962.268529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 968.310712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 968.329037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.746366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.427924] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 979.229925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 979.477255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.764365] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 986.551495] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 989.260463] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 995.590183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 995.990536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.731789] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1005.171987] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1015.473125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1020.900898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1020.907931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.932985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.937901] Lustre: Skipped 1 previous similar message [ 1020.947687] Lustre: Skipped 1 previous similar message [ 1026.026232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1026.036361] Lustre: Skipped 2 previous similar messages [ 1026.317328] LustreError: 40219:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1026.467531] Lustre: server umount lustre-OST0000 complete [ 1046.497985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.507913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.521243] Lustre: Skipped 1 previous similar message [ 1050.659388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.670375] Lustre: Skipped 2 previous similar messages [ 1052.567617] LustreError: 40609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1052.580753] LustreError: 40609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1052.684319] Lustre: server umount lustre-MDT0000 complete [ 1055.717504] LustreError: 37945:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.747527] LustreError: 37945:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1056.521180] LustreError: 37924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246077 with bad export cookie 2737330377678768809 [ 1056.529811] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.554261] LustreError: 37924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1062.800264] LustreError: 40812:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1062.809954] LustreError: 40812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1062.998824] Lustre: server umount lustre-MDT0001 complete [ 1071.623813] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1073.899943] Key type lgssc unregistered [ 1074.163166] LNet: 41344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.170556] LNetError: 41344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.181520] LNet: Removed LNI 192.168.203.160@tcp [ 1074.844401] Key type .llcrypt unregistered [ 1074.846443] Key type ._llcrypt unregistered [ 1093.361268] Key type ._llcrypt registered [ 1093.365297] Key type .llcrypt registered [ 1093.483454] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1094.413676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.516961] alg: No test for adler32 (adler32-zlib) [ 1095.657609] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1095.902461] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1097.594048] Key type lgssc registered [ 1098.603745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.127727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.158771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.862620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.877695] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1118.060741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1118.178225] LustreError: 42718:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.198848] LustreError: 42718:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1118.415611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.011257] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1127.261428] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1130.679466] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1136.367143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1136.754586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.597422] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1145.966281] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1153.139428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1161.668502] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.60@tcp (stopping) [ 1163.202151] LustreError: 44985:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1163.302806] Lustre: server umount lustre-MDT0000 complete [ 1163.746287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.752403] LustreError: 42713:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.775469] Lustre: Skipped 1 previous similar message [ 1163.786020] LustreError: 42713:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1166.743142] LustreError: 42698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246188 with bad export cookie 7899207129489711338 [ 1166.759130] LustreError: 42713:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.762712] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.787649] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.60@tcp (stopping) [ 1166.879850] LustreError: 45187:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1166.883041] LustreError: 45187:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1167.150621] Lustre: server umount lustre-MDT0001 complete [ 1179.457403] LustreError: 42698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.60@tcp arrived at 1761246200 with bad export cookie 7899207129489712003 [ 1179.474556] LustreError: 42698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1198.661718] LustreError: 45410:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1198.674897] LustreError: 45410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1198.863878] Lustre: server umount lustre-OST0000 complete [ 1212.348828] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1214.636535] Key type lgssc unregistered [ 1214.894990] LNet: 46249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.902651] LNetError: 46249:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.913525] LNet: Removed LNI 192.168.203.160@tcp [ 1215.696715] Key type .llcrypt unregistered [ 1215.698691] Key type ._llcrypt unregistered [ 1235.362586] Key type ._llcrypt registered [ 1235.366392] Key type .llcrypt registered [ 1235.489638] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1236.236995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1236.271611] alg: No test for adler32 (adler32-zlib) [ 1237.292098] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1237.480680] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1239.121143] Key type lgssc registered [ 1240.197641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1246.998897] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1247.023294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1262.496309] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1267.616295] LustreError: 47594:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 1294.304208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1299.662683] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1304.939321] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1471.446182] LustreError: 48302:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1471.592186] Lustre: server umount lustre-OST0000 complete [ 1485.150646] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1487.576819] Key type lgssc unregistered [ 1487.861563] LNet: 49140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.874505] LNetError: 49140:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.887043] LNet: Removed LNI 192.168.203.160@tcp [ 1488.611327] Key type .llcrypt unregistered [ 1488.615364] Key type ._llcrypt unregistered [ 1507.098798] Key type ._llcrypt registered [ 1507.100842] Key type .llcrypt registered [ 1507.187309] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1508.285030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1508.347502] alg: No test for adler32 (adler32-zlib) [ 1509.515530] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1509.722886] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1511.440170] Key type lgssc registered [ 1512.521635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1519.328626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1519.341095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.677664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1523.810336] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1530.851855] LustreError: 50513:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.880383] LustreError: 50513:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1531.573087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1531.864304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.360471] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1539.869913] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1543.361995] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1550.442562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1550.703062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.416698] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1561.674986] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1566.116984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1566.121352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.118206] LustreError: 52760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1570.377684] Lustre: server umount lustre-OST0000 complete [ 1576.424206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.431569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.462356] Lustre: Skipped 1 previous similar message [ 1576.465474] Lustre: Skipped 1 previous similar message [ 1580.655635] LustreError: 52962:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1580.672429] LustreError: 52962:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1580.847859] Lustre: server umount lustre-MDT0000 complete [ 1581.538900] LustreError: 52411:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.562914] LustreError: 52411:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1584.961292] LustreError: 50495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246606 with bad export cookie 14033128366117503885 [ 1584.964667] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1584.973289] LustreError: 50495:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1585.147483] LustreError: 53164:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1585.172299] LustreError: 53164:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1585.449765] Lustre: server umount lustre-MDT0001 complete [ 1594.792704] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1597.229604] Key type lgssc unregistered [ 1597.499058] LNet: 53695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.509409] LNetError: 53695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.525871] LNet: Removed LNI 192.168.203.160@tcp [ 1598.314658] Key type .llcrypt unregistered [ 1598.316529] Key type ._llcrypt unregistered [ 1616.871703] Key type ._llcrypt registered [ 1616.874608] Key type .llcrypt registered [ 1616.953871] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1617.564641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.588280] alg: No test for adler32 (adler32-zlib) [ 1618.646524] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1618.875535] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1620.520211] Key type lgssc registered [ 1621.530822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1628.367872] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.382191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1644.000434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1649.120751] LustreError: 55041:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 1675.808225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.265455] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1684.520043] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1690.491561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.330586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.403886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1695.188226] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1696.751546] LustreError: 55067:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.768445] LustreError: 55067:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1701.857732] LustreError: 55067:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.651439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1703.065409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1706.146762] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1711.303246] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1714.280942] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1717.217722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1717.222331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.237365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.241874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1718.247632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1718.267958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.806220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.909170] LustreError: 57321:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1722.014325] Lustre: server umount lustre-OST0000 complete [ 1743.841755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1743.850810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.878079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.880451] Lustre: Skipped 1 previous similar message [ 1745.955834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.134055] LustreError: 57699:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1748.138378] LustreError: 57699:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1748.242583] Lustre: server umount lustre-MDT0000 complete [ 1751.009324] LustreError: 55968:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.024333] LustreError: 55968:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1751.526386] LustreError: 55995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246772 with bad export cookie 5838979933063727679 [ 1751.534403] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1751.534662] LustreError: 55995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1757.754329] LustreError: 57900:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1757.757721] LustreError: 57900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1758.016582] Lustre: server umount lustre-MDT0001 complete [ 1767.340493] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1769.602908] Key type lgssc unregistered [ 1769.807118] LNet: 58433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.814531] LNetError: 58433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.829334] LNet: Removed LNI 192.168.203.160@tcp [ 1770.361772] Key type .llcrypt unregistered [ 1770.368302] Key type ._llcrypt unregistered [ 1788.481697] Key type ._llcrypt registered [ 1788.483337] Key type .llcrypt registered [ 1788.591035] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1789.420391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.456755] alg: No test for adler32 (adler32-zlib) [ 1790.494929] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1790.698235] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1792.352201] Key type lgssc registered [ 1793.471701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.814417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.838506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.230255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.456436] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1811.429784] LustreError: 59810:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.461043] LustreError: 59810:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1812.019984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1812.416276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.628992] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1820.057846] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1823.179996] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1828.163506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1828.488732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.611307] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1838.380565] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1845.935702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1856.488026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.488771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.496563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.510254] Lustre: Skipped 1 previous similar message [ 1859.207555] LustreError: 62078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1859.331614] Lustre: server umount lustre-OST0000 complete [ 1863.649370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1863.658338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.666248] Lustre: Skipped 1 previous similar message [ 1863.671187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.722613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.726432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.080503] LustreError: 62281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1868.088174] LustreError: 62281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1868.131160] Lustre: server umount lustre-MDT0000 complete [ 1871.535257] LustreError: 59792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246892 with bad export cookie 12248063926715276111 [ 1871.538494] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.560138] LustreError: 59792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1871.698443] LustreError: 62483:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1871.702781] LustreError: 62483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1872.027361] Lustre: server umount lustre-MDT0001 complete [ 1880.959442] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 1883.484498] Key type lgssc unregistered [ 1883.738765] LNet: 63014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.752451] LNetError: 63014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.782721] LNet: Removed LNI 192.168.203.160@tcp [ 1884.515057] Key type .llcrypt unregistered [ 1884.518630] Key type ._llcrypt unregistered [ 1913.379444] Key type ._llcrypt registered [ 1913.381593] Key type .llcrypt registered [ 1913.509520] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 1914.498582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.545094] alg: No test for adler32 (adler32-zlib) [ 1915.555967] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1915.788779] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 1917.448638] Key type lgssc registered [ 1918.200279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.110293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.135553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.711359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.277125] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1936.867379] LustreError: 64678:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.883588] LustreError: 64678:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1937.860990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1938.165204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1941.703308] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1946.414473] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.275075] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1955.201658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1955.478811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.765494] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1967.353757] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1971.968409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1976.724643] Lustre: Failing over lustre-MDT0000 [ 1976.849560] LustreError: 66924:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1976.905506] Lustre: server umount lustre-MDT0000 complete [ 1977.315126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.317657] LustreError: 65392:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.342234] Lustre: Skipped 2 previous similar messages [ 1977.359890] LustreError: 65392:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1982.437525] LustreError: 65739:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.444330] LustreError: 65739:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1984.974994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.062445] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1985.183331] Lustre: *** cfs_fail_loc=135, val=0*** [ 1985.269550] LustreError: 67434:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1985.274899] LustreError: 67434:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.160@tcp: cfg command failed: rc = -2 [ 1985.280336] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1985.280336] [ 1985.286543] LustreError: MGC192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1985.299263] LustreError: 67421:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1985.306302] LustreError: 67421:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 1985.317069] LustreError: 67421:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1985.340979] Lustre: server umount lustre-MDT0000 complete [ 1985.343388] LustreError: 67421:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1987.555177] LustreError: 64679:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.567353] LustreError: 64679:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1990.372375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.447534] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1990.593479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.630678] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1993.388984] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 1993.640587] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1995.755923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1995.771941] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1995.800225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1998.817030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1998.819625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.840244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.866292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.869611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2000.870535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.655989] LustreError: 68244:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2004.658982] LustreError: 68244:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2004.815596] Lustre: server umount lustre-OST0000 complete [ 2011.110958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.112281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.118809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.123527] Lustre: Skipped 1 previous similar message [ 2011.134905] Lustre: Skipped 2 previous similar messages [ 2016.228368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.345357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.354278] Lustre: Skipped 2 previous similar messages [ 2022.369545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2022.493025] LustreError: 68446:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2022.496660] LustreError: 68446:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2022.597352] Lustre: server umount lustre-MDT0000 complete [ 2025.845772] LustreError: 66925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247047 with bad export cookie 15565108569879952893 [ 2025.850777] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.852325] LustreError: 66925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2025.956517] LustreError: 68648:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2025.961228] LustreError: 68648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2026.169485] Lustre: server umount lustre-MDT0001 complete [ 2034.899243] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2036.948236] Key type lgssc unregistered [ 2037.181851] LNet: 69180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.186995] LNetError: 69180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.247263] LNet: Removed LNI 192.168.203.160@tcp [ 2038.836456] Key type .llcrypt unregistered [ 2038.839752] Key type ._llcrypt unregistered [ 2055.704839] Key type ._llcrypt registered [ 2055.706724] Key type .llcrypt registered [ 2055.797410] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2056.495883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.650324] alg: No test for adler32 (adler32-zlib) [ 2057.711124] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2057.963226] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2059.656164] Key type lgssc registered [ 2060.547763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.313489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2067.336295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.919192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.150670] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2079.202608] LustreError: 70553:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.238574] LustreError: 70553:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2081.065146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2081.352562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.727711] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2089.738136] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2092.847238] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2098.299663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2098.575724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.046152] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2107.165390] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2113.904187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2116.984757] Lustre: Failing over lustre-MDT0000 [ 2117.093085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2117.095554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.103788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.125857] LustreError: 72806:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2117.190948] Lustre: server umount lustre-MDT0000 complete [ 2119.088981] LustreError: 70549:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.142372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.152479] Lustre: Skipped 1 previous similar message [ 2124.197174] LustreError: 70550:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.223341] LustreError: 70550:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2124.503337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.577781] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.665289] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2124.748565] LustreError: 73315:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2124.753132] LustreError: 73315:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.160@tcp: cfg command failed: rc = -9 [ 2124.758138] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2124.758138] [ 2124.766140] LustreError: MGC192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2124.777984] LustreError: 73303:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2124.788924] LustreError: 73303:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -9 [ 2124.807279] LustreError: 73303:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2124.825763] Lustre: server umount lustre-MDT0000 complete [ 2124.828065] LustreError: 73303:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2129.321602] LustreError: 72544:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.337431] LustreError: 72544:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2130.551790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.660854] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.863877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.930687] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2134.033993] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2134.441151] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2136.047289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2136.065268] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2136.123730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2141.159492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.168871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.193198] Lustre: Skipped 1 previous similar message [ 2142.462337] LustreError: 74126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2142.465374] LustreError: 74126:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2142.611458] Lustre: server umount lustre-OST0000 complete [ 2146.273799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2146.277206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.287275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.320722] Lustre: Skipped 1 previous similar message [ 2146.326056] Lustre: Skipped 1 previous similar message [ 2151.395103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.407159] Lustre: Skipped 1 previous similar message [ 2152.097698] LustreError: 74328:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2152.101036] LustreError: 74328:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2152.197285] Lustre: server umount lustre-MDT0000 complete [ 2155.680362] LustreError: 70534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247177 with bad export cookie 5194742069478857640 [ 2155.686988] LustreError: 70534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2155.687930] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.924715] LustreError: 74530:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2155.932477] LustreError: 74530:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2156.155502] Lustre: server umount lustre-MDT0001 complete [ 2166.116694] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2168.527670] Key type lgssc unregistered [ 2168.789732] LNet: 75061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.799133] LNetError: 75061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.821108] LNet: Removed LNI 192.168.203.160@tcp [ 2169.586342] Key type .llcrypt unregistered [ 2169.588231] Key type ._llcrypt unregistered [ 2180.110256] Key type ._llcrypt registered [ 2180.111796] Key type .llcrypt registered [ 2180.228474] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2180.486383] Key type .llcrypt unregistered [ 2180.492339] Key type ._llcrypt unregistered [ 2200.012832] Key type ._llcrypt registered [ 2200.014454] Key type .llcrypt registered [ 2200.118444] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2200.822778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.829599] alg: No test for adler32 (adler32-zlib) [ 2201.857491] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2202.048547] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2203.704176] Key type lgssc registered [ 2204.697942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.316207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2211.351126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.800269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.863413] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2222.380795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2222.736701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2225.817532] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2231.074152] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2234.360908] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2240.789607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2241.166812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.317681] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2251.649343] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2258.610756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2269.154722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2269.155774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.166583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2269.168268] Lustre: Skipped 1 previous similar message [ 2269.173852] Lustre: Skipped 1 previous similar message [ 2274.281590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2274.284513] Lustre: Skipped 1 previous similar message [ 2275.032184] LustreError: 79172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2275.128635] Lustre: server umount lustre-OST0000 complete [ 2278.881669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2278.885592] LustreError: Skipped 1 previous similar message [ 2278.888401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.895573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2283.737428] LustreError: 79374:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2283.740702] LustreError: 79374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2283.827713] Lustre: server umount lustre-MDT0000 complete [ 2284.513634] LustreError: 76899:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.521455] LustreError: 76899:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2286.348449] LustreError: 76882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247307 with bad export cookie 3434127008030237231 [ 2286.349137] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.356903] LustreError: 76882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2286.452392] LustreError: 79577:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2286.490793] LustreError: 79577:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2286.666380] Lustre: server umount lustre-MDT0001 complete [ 2295.211727] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2297.356842] Key type lgssc unregistered [ 2297.701607] LNet: 80108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.717838] LNetError: 80108:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.733380] LNet: Removed LNI 192.168.203.160@tcp [ 2298.361661] Key type .llcrypt unregistered [ 2298.364755] Key type ._llcrypt unregistered [ 2316.739399] Key type ._llcrypt registered [ 2316.744106] Key type .llcrypt registered [ 2316.831293] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2317.847376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.140844] alg: No test for adler32 (adler32-zlib) [ 2319.208385] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2319.395729] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2321.080775] Key type lgssc registered [ 2322.260766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2329.865066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2329.882411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.350953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2335.460330] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2340.843295] LustreError: 81480:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.861382] LustreError: 81480:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2342.959499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2343.322340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2347.117815] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2352.425244] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2356.240645] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2361.672943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2361.937608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.815564] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2369.704414] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2376.310471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2386.913205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.923489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.925920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.946682] Lustre: Skipped 1 previous similar message [ 2390.788120] LustreError: 83748:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2390.947969] Lustre: server umount lustre-OST0000 complete [ 2394.595408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2394.599062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.631108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2394.638872] Lustre: Skipped 1 previous similar message [ 2397.154030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.158598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.168110] Lustre: Skipped 1 previous similar message [ 2400.124413] LustreError: 83950:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2400.135928] LustreError: 83950:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2400.234149] Lustre: server umount lustre-MDT0000 complete [ 2402.273524] LustreError: 82511:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.287808] LustreError: 82511:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2403.613706] LustreError: 81462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247425 with bad export cookie 15023868135564651098 [ 2403.615548] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.628505] LustreError: 81462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2403.779785] LustreError: 84152:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2403.790175] LustreError: 84152:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2404.072336] Lustre: server umount lustre-MDT0001 complete [ 2414.613430] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2416.807721] Key type lgssc unregistered [ 2417.051987] LNet: 84686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2417.060120] LNetError: 84686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.070742] LNet: Removed LNI 192.168.203.160@tcp [ 2417.686182] Key type .llcrypt unregistered [ 2417.687807] Key type ._llcrypt unregistered [ 2436.292723] Key type ._llcrypt registered [ 2436.295018] Key type .llcrypt registered [ 2436.420807] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2437.370624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.386724] alg: No test for adler32 (adler32-zlib) [ 2438.408674] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2438.712105] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2440.387231] Key type lgssc registered [ 2441.469905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2448.130775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2448.150280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.554044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.742722] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2459.105890] LustreError: 86061:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.115361] LustreError: 86061:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2459.658694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2459.991333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.534706] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2468.493787] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2472.457316] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2479.061260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2479.478039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.129396] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2489.937996] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2493.809098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2504.161830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.163700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.179368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.182399] Lustre: Skipped 1 previous similar message [ 2505.466104] LustreError: 88329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2505.590531] Lustre: server umount lustre-OST0000 complete [ 2509.283294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.303888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2509.306896] Lustre: Skipped 1 previous similar message [ 2514.402254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.408227] Lustre: Skipped 2 previous similar messages [ 2514.534607] LustreError: 88531:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2514.539126] LustreError: 88531:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2514.682039] Lustre: server umount lustre-MDT0000 complete [ 2517.907695] LustreError: 86040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247539 with bad export cookie 16611565523569417632 [ 2517.913289] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.916963] LustreError: 86040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2518.035907] LustreError: 88733:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2518.042527] LustreError: 88733:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2518.288295] Lustre: server umount lustre-MDT0001 complete [ 2527.183367] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2529.485554] Key type lgssc unregistered [ 2529.715532] LNet: 89265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.726364] LNetError: 89265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.791911] LNet: Removed LNI 192.168.203.160@tcp [ 2531.511527] Key type .llcrypt unregistered [ 2531.518687] Key type ._llcrypt unregistered [ 2550.727969] Key type ._llcrypt registered [ 2550.729662] Key type .llcrypt registered [ 2550.844975] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2551.647323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.665946] alg: No test for adler32 (adler32-zlib) [ 2552.670071] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2552.891305] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2554.568206] Key type lgssc registered [ 2555.581708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2562.386750] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.405753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2577.953390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2583.074548] LustreError: 90610:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 2609.764407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.997806] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2620.173921] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2625.044080] LustreError: 91466:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2625.181706] Lustre: server umount lustre-OST0000 complete [ 2632.718503] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 15:27:32 (1761247652) [ 2640.097631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.585309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.578861] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2646.003321] LustreError: 92135:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.011853] LustreError: 92135:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2650.086505] LustreError: 92135:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.856232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2653.254071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.739877] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2662.247613] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2665.463534] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2672.090301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2672.387449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.076945] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2681.643762] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2684.659419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2699.237218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.243271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.251928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.261919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2700.265491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.267873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.298166] Lustre: Skipped 1 previous similar message [ 2702.579835] LustreError: 94542:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2702.595444] LustreError: 94542:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2702.933719] Lustre: server umount lustre-OST0000 complete [ 2709.474586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.478542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.491354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.331129] LustreError: 94743:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2713.338897] LustreError: 94743:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2713.452701] Lustre: server umount lustre-MDT0000 complete [ 2715.618707] LustreError: 92130:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.628389] LustreError: 92130:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2716.950520] LustreError: 92117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247738 with bad export cookie 11613543551821701752 [ 2716.955729] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.147843] LustreError: 94946:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2717.157503] LustreError: 94946:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2717.368811] Lustre: server umount lustre-MDT0001 complete [ 2725.783608] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2728.334334] Key type lgssc unregistered [ 2728.655667] LNet: 95477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.673740] LNetError: 95477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2728.693044] LNet: Removed LNI 192.168.203.160@tcp [ 2729.412926] Key type .llcrypt unregistered [ 2729.414711] Key type ._llcrypt unregistered [ 2749.664955] Key type ._llcrypt registered [ 2749.666657] Key type .llcrypt registered [ 2749.773715] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2750.625330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2750.683683] alg: No test for adler32 (adler32-zlib) [ 2751.802111] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2752.103489] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2753.744251] Key type lgssc registered [ 2755.020724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2762.906447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2762.933614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.438252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.172877] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2773.988992] LustreError: 96850:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.011771] LustreError: 96850:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2776.348287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2776.715594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2780.455504] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2786.135725] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2789.436581] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2797.191835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2797.553501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.678297] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2807.990472] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2808.828961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2822.115519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2822.128487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.166041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.168399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.172692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2824.198217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.185167] LustreError: 99118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2826.449522] Lustre: server umount lustre-OST0000 complete [ 2832.865703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2832.869047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2832.907260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.911508] Lustre: Skipped 1 previous similar message [ 2836.380567] LustreError: 99319:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2836.390154] LustreError: 99319:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2836.499752] Lustre: server umount lustre-MDT0000 complete [ 2839.520910] LustreError: 96851:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.531285] LustreError: 96851:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2840.175057] LustreError: 96831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247861 with bad export cookie 12625797046745005291 [ 2840.184987] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2840.193944] LustreError: 96831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2840.404425] LustreError: 99521:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2840.458233] LustreError: 99521:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2840.675878] Lustre: server umount lustre-MDT0001 complete [ 2850.527817] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 2853.097685] Key type lgssc unregistered [ 2853.431565] LNet: 100053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2853.463181] LNetError: 100053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.505059] LNet: Removed LNI 192.168.203.160@tcp [ 2855.359425] Key type .llcrypt unregistered [ 2855.365066] Key type ._llcrypt unregistered [ 2872.515699] Key type ._llcrypt registered [ 2872.518114] Key type .llcrypt registered [ 2872.596977] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2873.280335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2873.331712] alg: No test for adler32 (adler32-zlib) [ 2874.395177] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2874.629397] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 2876.312192] Key type lgssc registered [ 2877.490863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2884.461163] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2884.492489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2900.064510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2905.184259] LustreError: 101276:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 2931.872299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.160886] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 2942.043947] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2948.277599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.618616] LustreError: MGC192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2948.635602] LustreError: 102187:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2948.661298] LustreError: 102187:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 2948.679875] LustreError: 102187:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2948.684946] LustreError: 102216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247970 with bad export cookie 14903612727056523058 [ 2948.746157] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2948.825313] Lustre: server umount lustre-MDT0000 complete [ 2948.827457] LustreError: 102187:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2955.992105] LustreError: 102620:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2956.272612] Lustre: server umount lustre-OST0000 complete [ 2965.255455] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 2974.428146] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 2983.986479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2991.654527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2998.815700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3004.781103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.984937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.043641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.268579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3010.296383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3010.359416] Lustre: lustre-MDT0000: new disk, initializing [ 3010.442121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.464628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3013.918835] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3023.418108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3023.479349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3023.567485] Lustre: 104956:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3023.626065] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3023.630783] Lustre: Skipped 1 previous similar message [ 3023.692380] Lustre: lustre-MDT0001: new disk, initializing [ 3023.827797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.879066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3023.900311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3023.972891] Lustre: 105752:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3023.980932] Lustre: 105752:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3023.987110] Lustre: 105752:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3023.997938] Lustre: 105752:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3024.003311] Lustre: 105752:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3024.017641] Lustre: 105752:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3027.903714] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3034.921476] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3038.258945] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3044.491654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3044.579630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3044.787047] Lustre: lustre-OST0000: new disk, initializing [ 3044.790762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3044.869739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3050.126150] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3054.693407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3054.714042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3054.756676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3057.133181] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.933871] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3061.177699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3064.798775] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3065.009768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3070.441435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3070.442047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.449384] LustreError: Skipped 1 previous similar message [ 3070.460844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3072.696217] LustreError: 107898:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3072.698689] LustreError: 107898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3072.868534] Lustre: server umount lustre-OST0000 complete [ 3080.162330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3080.165886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.186114] Lustre: Skipped 1 previous similar message [ 3080.194817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.200512] Lustre: Skipped 1 previous similar message [ 3082.347780] LustreError: 108100:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3082.356640] LustreError: 108100:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3082.444950] Lustre: server umount lustre-MDT0000 complete [ 3085.626206] LustreError: 106705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248107 with bad export cookie 14903612727056523142 [ 3085.631909] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.632777] LustreError: 106705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3085.762179] LustreError: 108301:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3085.764542] LustreError: 108301:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3085.988493] Lustre: server umount lustre-MDT0001 complete [ 3092.319928] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 15:35:12 (1761248112) [ 3095.121846] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3107.740981] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 3114.832502] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3121.881259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.289094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.752823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3138.345341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3143.260562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3143.351221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3143.619622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3143.664289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3143.777124] Lustre: lustre-MDT0000: new disk, initializing [ 3143.878037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.909612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3146.981931] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3155.787381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.846508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.922765] Lustre: 111436:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3155.972803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3155.975848] Lustre: Skipped 1 previous similar message [ 3156.045824] Lustre: lustre-MDT0001: new disk, initializing [ 3156.121021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3156.159019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3156.170633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3159.210181] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3166.011478] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3169.376364] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3175.160042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3175.237442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3175.487343] Lustre: lustre-OST0000: new disk, initializing [ 3175.491772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3175.570072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.417776] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3185.179080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3185.190165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3185.271588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3186.455845] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3189.193870] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3189.354212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3192.079392] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3192.271962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3195.374982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3195.375408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.379026] LustreError: Skipped 1 previous similar message [ 3195.379686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3195.393749] Lustre: Skipped 2 previous similar messages [ 3195.404188] Lustre: Skipped 2 previous similar messages [ 3199.740982] LustreError: 114376:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3199.743789] LustreError: 114376:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3199.909794] Lustre: server umount lustre-OST0000 complete [ 3205.602599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.608770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.625467] Lustre: Skipped 1 previous similar message [ 3209.472216] LustreError: 114577:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3209.485377] LustreError: 114577:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3209.553640] Lustre: server umount lustre-MDT0000 complete [ 3212.958468] LustreError: 111429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248234 with bad export cookie 14903612727056524073 [ 3212.964351] LustreError: 111429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3212.969605] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.310494] Lustre: server umount lustre-MDT0001 complete [ 3221.503139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.853777] LustreError: 115256:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.877575] LustreError: 115256:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 3221.953549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.873708] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3227.116848] LustreError: 115256:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.245529] LustreError: 115256:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.652940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3236.047270] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3240.968944] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3244.489957] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3250.472474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3255.781071] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3260.634845] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3275.985157] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3278.309116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.311167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.316228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.323238] Lustre: Skipped 1 previous similar message [ 3278.331568] Lustre: Skipped 1 previous similar message [ 3284.305428] LustreError: 117618:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3284.308764] LustreError: 117618:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3284.500673] Lustre: server umount lustre-OST0000 complete [ 3288.556448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.578226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.580734] Lustre: Skipped 3 previous similar messages [ 3294.029248] Lustre: server umount lustre-MDT0000 complete [ 3296.813717] LustreError: 115237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248318 with bad export cookie 14903612727056524997 [ 3296.817887] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3296.836336] LustreError: 115237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3297.142652] Lustre: server umount lustre-MDT0001 complete [ 3304.603163] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 3306.792386] Key type lgssc unregistered [ 3307.042682] LNet: 118556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3307.053360] LNetError: 118556:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.071497] LNet: Removed LNI 192.168.203.160@tcp [ 3308.828091] Key type .llcrypt unregistered [ 3308.844253] Key type ._llcrypt unregistered [ 3323.443099] Key type ._llcrypt registered [ 3323.446517] Key type .llcrypt registered [ 3323.551128] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 3335.666623] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 3336.344908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3336.362950] alg: No test for adler32 (adler32-zlib) [ 3337.500506] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3337.711253] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 3339.400158] Key type lgssc registered [ 3340.476531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3349.133401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3355.259839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.412537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3365.995354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3371.599716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3371.658141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3371.675604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.834098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3372.860593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3372.931773] Lustre: lustre-MDT0000: new disk, initializing [ 3373.047110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.062970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3376.384933] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3385.791181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3385.878981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3386.008060] Lustre: 121661:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3386.087834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3386.101262] Lustre: Skipped 1 previous similar message [ 3386.205903] Lustre: lustre-MDT0001: new disk, initializing [ 3386.328041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3386.395077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3386.403258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3390.052862] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3396.662992] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3399.484453] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3405.494673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3405.593468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3405.822378] Lustre: lustre-OST0000: new disk, initializing [ 3405.829890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3405.939564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.017213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3411.038883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3411.168577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3411.176807] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3418.313028] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3422.614035] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3422.861088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3426.376703] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3426.597631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3431.906310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3431.907180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.929024] LustreError: Skipped 1 previous similar message [ 3431.937427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3434.440894] LustreError: 124604:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3434.597710] Lustre: server umount lustre-OST0000 complete [ 3442.148071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.152093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3442.159048] Lustre: Skipped 2 previous similar messages [ 3442.170756] Lustre: Skipped 2 previous similar messages [ 3447.267544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3447.277208] Lustre: Skipped 1 previous similar message [ 3452.385925] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.390272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3452.400593] Lustre: Skipped 1 previous similar message [ 3452.473298] LustreError: 124806:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3452.483694] LustreError: 124806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3452.616375] Lustre: server umount lustre-MDT0000 complete [ 3456.386321] LustreError: 122443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248477 with bad export cookie 14556915527674937040 [ 3456.388770] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3456.399506] LustreError: 122443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3456.529833] LustreError: 125010:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3456.543616] LustreError: 125010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3456.774123] Lustre: server umount lustre-MDT0001 complete [ 3465.087306] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:41:24 (1761248484) [ 3470.742598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.188231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.396221] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3476.451067] LustreError: 125683:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3476.462374] LustreError: 125683:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3481.582779] LustreError: 125683:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.150376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3482.507022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3486.228650] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3491.045120] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3494.331486] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3495.977800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.012895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3502.195410] LustreError: 127026:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3502.202683] LustreError: 127026:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3502.267157] Lustre: server umount lustre-MDT0000 complete [ 3505.667178] LustreError: 125663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248527 with bad export cookie 14556915527674937964 [ 3505.674056] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.680787] LustreError: 125663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3506.020648] Lustre: server umount lustre-MDT0001 complete [ 3513.393783] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:42:13 (1761248533) [ 3519.264097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3534.816600] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3539.936440] LustreError: 127873:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 3565.600267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.759103] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3573.407855] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3574.686592] LustreError: 128537:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3574.693751] LustreError: 128537:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3574.838340] Lustre: server umount lustre-OST0000 complete [ 3580.114344] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:43:20 (1761248600) [ 3585.131107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.463496] LustreError: 129208:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3585.534137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.260644] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3590.635356] LustreError: 129208:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.735859] LustreError: 129208:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.974754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3599.543892] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3604.061270] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3608.288801] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3614.827315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3620.387101] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3625.914488] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3636.708803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.711031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.728085] Lustre: Skipped 3 previous similar messages [ 3636.731089] Lustre: Skipped 4 previous similar messages [ 3642.876235] LustreError: 131463:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3642.898390] LustreError: 131463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3643.014474] Lustre: server umount lustre-MDT0000 complete [ 3646.680956] LustreError: 129188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248668 with bad export cookie 14556915527674938629 [ 3646.689224] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3646.704788] LustreError: 129188:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3646.949880] LustreError: 129202:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.950510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.193128] Lustre: server umount lustre-MDT0001 complete [ 3652.959199] Lustre: server umount lustre-OST0000 complete [ 3659.962400] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:44:39 (1761248679) [ 3665.880961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.157850] LustreError: 132537:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.181215] LustreError: 132537:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3666.222798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.225919] Lustre: Skipped 2 previous similar messages [ 3670.208376] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3678.110671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3682.344251] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3688.879843] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3692.901303] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3699.536405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3704.761034] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3709.720510] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3711.161039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3713.026484] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3713.215758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3716.771304] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3716.968448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3721.700919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3721.710180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.723289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.741978] Lustre: Skipped 4 previous similar messages [ 3724.728985] LustreError: 135285:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3724.744488] LustreError: 135285:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3724.918292] Lustre: server umount lustre-OST0000 complete [ 3729.892710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3742.178067] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3742.365672] Lustre: server umount lustre-MDT0000 complete [ 3745.556127] LustreError: 135488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248766 with bad export cookie 14556915527674939700 [ 3745.563397] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3745.595100] LustreError: 135488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3745.921666] Lustre: server umount lustre-MDT0001 complete [ 3753.800291] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:46:13 (1761248773) [ 3759.976266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3775.584825] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3780.704632] LustreError: 136334:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 3806.432464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.436376] Lustre: Skipped 2 previous similar messages [ 3810.878545] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3814.699889] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3820.195361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.540447] LustreError: 136361:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.553222] LustreError: 136361:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3820.663307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3823.754905] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3830.625487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3833.918799] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3838.168505] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3841.378384] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3844.783326] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3844.956348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3847.740428] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3847.928858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3851.234410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3851.242441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3851.254595] Lustre: Skipped 3 previous similar messages [ 3851.259943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.265969] Lustre: Skipped 7 previous similar messages [ 3855.619990] LustreError: 139113:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3855.624223] LustreError: 139113:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3855.735557] Lustre: server umount lustre-MDT0000 complete [ 3859.101554] LustreError: 136341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248880 with bad export cookie 14556915527674940722 [ 3859.107923] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3859.116663] LustreError: 136341:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3859.435943] Lustre: server umount lustre-MDT0001 complete [ 3873.801639] Lustre: server umount lustre-OST0000 complete [ 3880.618768] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:48:20 (1761248900) [ 3886.480944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3902.049215] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3907.168488] LustreError: 140161:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 3936.941987] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3941.685849] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3948.109880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.417476] LustreError: 140188:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.451630] LustreError: 140188:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 7 previous similar messages [ 3948.540424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.543396] Lustre: Skipped 3 previous similar messages [ 3948.632932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3952.459308] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3961.690374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3966.373585] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3971.965872] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3975.316648] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3980.997414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3981.080400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3981.228385] Lustre: lustre-OST0001: new disk, initializing [ 3981.234574] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3986.234410] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 3990.038873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3990.052308] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3990.144856] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3993.454312] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3997.994362] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3998.268584] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4001.758967] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4001.957163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4004.839048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4004.848997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.854235] Lustre: Skipped 2 previous similar messages [ 4004.858598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.861645] Lustre: Skipped 2 previous similar messages [ 4010.383489] LustreError: 143896:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4010.387796] LustreError: 143896:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4010.441492] Lustre: server umount lustre-OST0000 complete [ 4015.586627] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4020.442341] Lustre: server umount lustre-OST0001 complete [ 4030.632928] Lustre: server umount lustre-MDT0000 complete [ 4033.552519] LustreError: 140169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249054 with bad export cookie 14556915527674941667 [ 4033.562905] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4033.981361] Lustre: server umount lustre-MDT0001 complete [ 4039.335358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4042.975403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4046.706705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4049.999608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4055.980154] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:51:15 (1761249075) [ 4057.562700] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4059.159771] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4060.569039] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:51:20 (1761249080) [ 4066.512618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.727680] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4066.754877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4070.089670] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4077.061228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4077.127310] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4077.152366] Lustre: 146141:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4080.730732] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4084.992494] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4087.688926] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4093.242095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4093.445720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4098.125615] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4099.142026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4102.747659] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4106.055588] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4106.224993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4109.371293] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4109.606463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4114.407453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.407593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.415733] Lustre: Skipped 5 previous similar messages [ 4114.423151] LustreError: Skipped 1 previous similar message [ 4117.257598] Lustre: server umount lustre-OST0000 complete [ 4136.192977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4141.591325] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4146.309282] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4150.194325] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4153.905091] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.937055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.940519] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4153.958765] Lustre: Skipped 1 previous similar message [ 4154.822329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4158.251222] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4158.466784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4161.961417] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d37c878b800.ost_server_uuid 50 [ 4163.361792] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d37c878b800.ost_server_uuid in FULL state after 0 sec [ 4163.517471] LustreError: 146146:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.60@tcp) returned error from blocking AST (req@000000005618b241 x1846802759527296 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d4fae64a/0xca048d8bec6eda36 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.60@tcp remote: 0x5d0c8c1075dd01dd expref: 5 pid: 146148 timeout: 4263 lvb_type: 0 [ 4163.543099] LustreError: lustre-MDT0000: A client on nid 192.168.203.60@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4163.562863] LustreError: 146137:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.60@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d4fae64a/0xca048d8bec6eda36 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.60@tcp remote: 0x5d0c8c1075dd01dd expref: 6 pid: 146148 timeout: 0 lvb_type: 0 [ 4169.189752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4169.196401] Lustre: Skipped 9 previous similar messages [ 4172.800133] LustreError: 150723:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4172.806747] LustreError: 150723:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4172.966401] Lustre: server umount lustre-OST0000 complete [ 4182.257922] Lustre: server umount lustre-MDT0000 complete [ 4184.545627] LustreError: 146152:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4184.555446] LustreError: 146152:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 9 previous similar messages [ 4184.997264] LustreError: 149049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249206 with bad export cookie 14556915527674942843 [ 4185.006569] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.014727] LustreError: 149049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4199.393039] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4199.813099] Lustre: server umount lustre-MDT0001 complete [ 4210.385192] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4213.055454] Key type lgssc unregistered [ 4213.302767] LNet: 151659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4213.314630] LNetError: 151659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4213.333215] LNet: Removed LNI 192.168.203.160@tcp [ 4214.032444] Key type .llcrypt unregistered [ 4214.034988] Key type ._llcrypt unregistered [ 4233.521902] Key type ._llcrypt registered [ 4233.523475] Key type .llcrypt registered [ 4233.679035] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4234.674285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4234.727052] alg: No test for adler32 (adler32-zlib) [ 4236.095080] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4236.354298] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4238.096120] Key type lgssc registered [ 4239.351346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4247.352743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4247.375827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.828979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.589553] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4259.303378] LustreError: 153030:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.343480] LustreError: 153030:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4260.778489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4261.003711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4263.880855] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4267.744561] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4270.972350] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4277.173100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4277.568396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.550103] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4287.878158] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4293.948983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4298.148672] Lustre: Failing over lustre-MDT0000 [ 4298.303307] LustreError: 155278:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4298.424323] Lustre: server umount lustre-MDT0000 complete [ 4299.235874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.238980] LustreError: 153745:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.273823] Lustre: Skipped 1 previous similar message [ 4299.291206] LustreError: 153745:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4304.359995] LustreError: 153025:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.386744] LustreError: 153025:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4309.481109] LustreError: 153027:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.492338] LustreError: 153027:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4310.449564] LustreError: 153012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.60@tcp arrived at 1761249331 with bad export cookie 3591994249797478775 [ 4314.597531] LustreError: 153745:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.606891] Lustre: 152344:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249320/real 1761249320] req@00000000674ca991 x1846803700932224/t0(0) o400->MGC192.168.203.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761249336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4314.662578] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4316.619100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4324.841612] LustreError: 152340:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005bd1f40d x1846803700937088/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4324.865763] LustreError: 153031:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.894130] LustreError: 153031:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 7 previous similar messages [ 4325.205675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.300381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4329.300693] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4330.479425] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4330.482867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4331.429456] Lustre: lustre-MDT0000: Denying connection for new client 51f55a74-9ad8-41f8-9aa0-fdcb171ba01a (at 192.168.203.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4340.707967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.714809] Lustre: Skipped 1 previous similar message [ 4340.718100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.720704] Lustre: Skipped 1 previous similar message [ 4345.682078] LustreError: 156212:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4345.708538] LustreError: 156212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4345.967104] Lustre: server umount lustre-OST0000 complete [ 4349.526672] LustreError: 156413:0:(ldlm_lib.c:3048:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4349.533297] Lustre: 155778:0:(ldlm_lib.c:2451:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4349.538678] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4349.562076] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4349.568775] LustreError: 155778:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008be845ce x1846803700951808/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4349.581244] LustreError: 155778:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4349.593967] LustreError: 155778:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4349.603873] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 4349.786851] LustreError: 156413:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4349.797241] LustreError: 156413:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4349.918212] Lustre: server umount lustre-MDT0000 complete [ 4350.945120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4350.959714] Lustre: Skipped 1 previous similar message [ 4350.969346] LustreError: 155191:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.530316] LustreError: 153011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249374 with bad export cookie 3591994249797479104 [ 4353.536433] LustreError: 153011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4353.538045] LustreError: 152340:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4353.542267] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4355.689433] LustreError: 156617:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4355.714858] LustreError: 156617:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4355.916120] Lustre: server umount lustre-MDT0001 complete [ 4364.110695] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4366.391799] Key type lgssc unregistered [ 4366.668284] LNet: 157151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4366.679661] LNetError: 157151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4366.706675] LNet: Removed LNI 192.168.203.160@tcp [ 4367.433867] Key type .llcrypt unregistered [ 4367.440192] Key type ._llcrypt unregistered [ 4385.963520] Key type ._llcrypt registered [ 4385.965545] Key type .llcrypt registered [ 4386.053681] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4386.843194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4386.900701] alg: No test for adler32 (adler32-zlib) [ 4387.965737] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4388.228561] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4389.888173] Key type lgssc registered [ 4390.841393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4397.278557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4397.304201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.644941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.102558] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4408.810242] LustreError: 158522:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4408.858924] LustreError: 158522:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4411.013915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4411.512578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4415.695900] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4422.134598] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4425.404951] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4431.965707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4432.301960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.005138] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4442.692426] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4444.607127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4451.816245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4451.826405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.846233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.888239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.905514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.937932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.534930] LustreError: 160774:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4457.738218] Lustre: server umount lustre-OST0000 complete [ 4462.567645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.571554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.585147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.589088] Lustre: Skipped 1 previous similar message [ 4465.127376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4465.127859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.243897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.249838] Lustre: Skipped 2 previous similar messages [ 4475.360761] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4475.440782] LustreError: 160976:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4475.445971] LustreError: 160976:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4475.539380] Lustre: server umount lustre-MDT0000 complete [ 4478.306858] LustreError: 160775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249499 with bad export cookie 8756395693922694942 [ 4478.310385] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.319790] LustreError: 160775:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4478.459470] LustreError: 161178:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4478.465015] LustreError: 161178:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4478.668786] Lustre: server umount lustre-MDT0001 complete [ 4486.797712] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4489.037992] Key type lgssc unregistered [ 4489.276139] LNet: 161710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4489.284398] LNetError: 161710:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4490.342927] LNet: Removed LNI 192.168.203.160@tcp [ 4491.054960] Key type .llcrypt unregistered [ 4491.056797] Key type ._llcrypt unregistered [ 4516.122344] Key type ._llcrypt registered [ 4516.123685] Key type .llcrypt registered [ 4516.215649] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4516.919433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4516.950207] alg: No test for adler32 (adler32-zlib) [ 4518.019330] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4518.214838] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4519.888167] Key type lgssc registered [ 4521.387643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4528.583516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4528.608354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.062457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.872993] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4540.388761] LustreError: 163372:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.398901] LustreError: 163372:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4541.265974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4541.647923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4545.238382] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4550.162503] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4553.809265] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4559.533485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4559.879598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.046616] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4570.134754] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4575.285983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4580.322729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.328068] Lustre: Skipped 1 previous similar message [ 4580.331172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.083143] LustreError: 165620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4582.391463] Lustre: server umount lustre-OST0000 complete [ 4588.001626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4588.005507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.013073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4588.015796] Lustre: Skipped 1 previous similar message [ 4590.568073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.568360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.685127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4595.700812] Lustre: Skipped 2 previous similar messages [ 4599.777778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4599.785472] Lustre: Skipped 1 previous similar message [ 4600.288136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4600.345254] LustreError: 165820:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4600.352878] LustreError: 165820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4600.460846] Lustre: server umount lustre-MDT0000 complete [ 4603.451280] LustreError: 163350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249624 with bad export cookie 14513454780521403507 [ 4603.461082] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4603.466521] LustreError: 163350:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4603.558442] LustreError: 166023:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4603.560957] LustreError: 166023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4603.740593] Lustre: server umount lustre-MDT0001 complete [ 4612.851837] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4615.271844] Key type lgssc unregistered [ 4615.555403] LNet: 166555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4615.560650] LNetError: 166555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4616.616173] LNet: Removed LNI 192.168.203.160@tcp [ 4617.345568] Key type .llcrypt unregistered [ 4617.347434] Key type ._llcrypt unregistered [ 4637.882961] Key type ._llcrypt registered [ 4637.884350] Key type .llcrypt registered [ 4637.971237] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4638.625700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4638.683117] alg: No test for adler32 (adler32-zlib) [ 4639.722352] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4640.058545] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4641.736193] Key type lgssc registered [ 4642.732705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4651.931803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4651.951749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.370392] Lustre: *** cfs_fail_loc=135, val=0*** [ 4653.468769] LustreError: 168074:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4653.473906] LustreError: 168074:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.160@tcp: cfg command failed: rc = -2 [ 4653.479139] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4653.479139] [ 4653.485072] LustreError: MGC192.168.203.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4653.493443] LustreError: 168035:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4653.499408] LustreError: 168035:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 4653.504242] LustreError: 168035:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4653.511830] LustreError: 168035:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4653.631326] Lustre: server umount lustre-MDT0000 complete [ 4653.633675] LustreError: 168035:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4659.679806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.064230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.788835] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4670.437623] LustreError: 168446:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.448484] LustreError: 168446:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4671.837896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4672.201323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4675.659729] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4681.321739] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4685.092341] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4687.334313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4687.343695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.364363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.889126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.892384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.573521] LustreError: 169791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4693.584969] LustreError: 169791:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4693.675475] Lustre: server umount lustre-MDT0000 complete [ 4695.009574] LustreError: 169159:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.017151] LustreError: 169159:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4697.065959] LustreError: 169159:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.075671] LustreError: 169159:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4697.126558] LustreError: 168427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249718 with bad export cookie 16499488846183649091 [ 4697.139189] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4697.144310] LustreError: 168427:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4697.273035] LustreError: 169995:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4697.283159] LustreError: 169995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4697.533959] Lustre: server umount lustre-MDT0001 complete [ 4706.287214] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4708.611439] Key type lgssc unregistered [ 4708.880740] LNet: 170528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4708.886635] LNetError: 170528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4708.903876] LNet: Removed LNI 192.168.203.160@tcp [ 4709.663920] Key type .llcrypt unregistered [ 4709.666894] Key type ._llcrypt unregistered [ 4729.054785] Key type ._llcrypt registered [ 4729.056542] Key type .llcrypt registered [ 4729.174085] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4729.476291] Key type .llcrypt unregistered [ 4729.480644] Key type ._llcrypt unregistered [ 4747.248992] Key type ._llcrypt registered [ 4747.250815] Key type .llcrypt registered [ 4747.365308] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4748.729760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4748.791735] alg: No test for adler32 (adler32-zlib) [ 4749.905442] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4750.234911] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4751.952200] Key type lgssc registered [ 4753.491807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4761.306625] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4761.329663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4776.864355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4781.986211] LustreError: 172623:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.160@tcp: failed processing log, type 4: rc = -110 [ 4808.608175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.138596] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4819.750462] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4826.982451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4827.521542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.574698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4832.397299] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4832.739164] LustreError: 172649:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.754391] LustreError: 172649:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4837.868605] LustreError: 172650:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4841.072835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4841.410370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4845.606207] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4851.243772] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4855.197349] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4858.411225] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4872.161942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4872.167680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.188730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4873.697677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4873.710749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4873.735067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4876.537582] LustreError: 175354:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4876.682388] Lustre: server umount lustre-OST0000 complete [ 4882.401136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4882.410383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.423152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4882.426223] Lustre: Skipped 2 previous similar messages [ 4884.449591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.450236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4884.460323] Lustre: Skipped 1 previous similar message [ 4885.757846] LustreError: 175555:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4885.762671] LustreError: 175555:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4885.847157] Lustre: server umount lustre-MDT0000 complete [ 4889.134809] LustreError: 174957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249910 with bad export cookie 7572978679126189353 [ 4889.147625] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4889.156714] LustreError: 174957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4889.286717] LustreError: 175757:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4889.291335] LustreError: 175757:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4889.554967] Lustre: server umount lustre-MDT0001 complete [ 4898.121636] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 4900.311982] Key type lgssc unregistered [ 4900.567826] LNet: 176288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4900.575175] LNetError: 176288:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4900.590524] LNet: Removed LNI 192.168.203.160@tcp [ 4901.221135] Key type .llcrypt unregistered [ 4901.223196] Key type ._llcrypt unregistered [ 4920.733916] Key type ._llcrypt registered [ 4920.735653] Key type .llcrypt registered [ 4920.836165] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 4921.578970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4921.626257] alg: No test for adler32 (adler32-zlib) [ 4922.698879] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4922.903280] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 4924.560206] Key type lgssc registered [ 4925.683591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4932.052962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4932.069810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.442587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.174753] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4942.825794] LustreError: 177660:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.845209] LustreError: 177660:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4943.394766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4943.804438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4947.687869] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4952.825368] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4956.408753] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4961.763931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4962.168587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4967.047891] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 4972.016201] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.553496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4986.607252] Lustre: Failing over lustre-MDT0000 [ 4986.741652] LustreError: 179982:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4986.987821] Lustre: server umount lustre-MDT0000 complete [ 4988.368665] LustreError: 179825:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.901846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.914509] Lustre: Skipped 2 previous similar messages [ 4993.458444] LustreError: 177656:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.470273] LustreError: 177656:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4998.566748] LustreError: 179825:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.576436] LustreError: 179825:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5003.681446] LustreError: 177655:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.706642] LustreError: 177655:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5005.280718] Lustre: 176972:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761250010/real 1761250010] req@00000000a3b79415 x1846804421305472/t0(0) o400->MGC192.168.203.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761250026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5005.302261] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5005.516596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.892238] LustreError: 179825:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.916766] LustreError: 179825:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 12 previous similar messages [ 5015.528836] LustreError: 176968:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000040e76405 x1846804421310848/t0(0) o250->MGC192.168.203.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5015.959651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.017660] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5020.070617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5020.110740] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5021.166241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5021.178487] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5021.255686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5024.662389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5026.721200] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5036.004944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5036.007941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.024401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.500283] LustreError: 181123:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5037.506466] LustreError: 181123:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5037.617385] Lustre: server umount lustre-OST0000 complete [ 5041.633550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.633964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5041.653355] Lustre: Skipped 1 previous similar message [ 5041.741562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.741569] Lustre: Skipped 1 previous similar message [ 5046.753834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.756754] Lustre: Skipped 1 previous similar message [ 5046.969250] LustreError: 181324:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5046.990391] LustreError: 181324:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5047.101058] Lustre: server umount lustre-MDT0000 complete [ 5051.035756] LustreError: 181018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250072 with bad export cookie 2837585286950013135 [ 5051.036529] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.045246] LustreError: 181018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5051.163110] LustreError: 181525:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5051.165493] LustreError: 181525:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5051.352544] Lustre: server umount lustre-MDT0001 complete [ 5060.077915] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5063.034405] Key type lgssc unregistered [ 5063.310313] LNet: 182058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5063.323583] LNetError: 182058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5063.343030] LNet: Removed LNI 192.168.203.160@tcp [ 5064.048508] Key type .llcrypt unregistered [ 5064.053289] Key type ._llcrypt unregistered [ 5084.203327] Key type ._llcrypt registered [ 5084.204700] Key type .llcrypt registered [ 5084.310403] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5084.995054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5085.049748] alg: No test for adler32 (adler32-zlib) [ 5086.073553] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5086.251847] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5087.896404] Key type lgssc registered [ 5088.825579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5096.096860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5096.124484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.551613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5100.468226] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5106.658942] LustreError: 183430:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.668092] LustreError: 183430:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5107.371367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5107.743373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5111.275493] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5116.425808] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5119.659131] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5125.849766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5126.238250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.799102] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5135.698362] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5141.631286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5148.463813] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5151.559771] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5163.809798] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5172.712969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5172.715565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5172.725831] Lustre: Skipped 1 previous similar message [ 5176.140177] LustreError: 185883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5176.358566] Lustre: server umount lustre-OST0000 complete [ 5182.953402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.958568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5182.973824] Lustre: Skipped 1 previous similar message [ 5182.982252] Lustre: Skipped 2 previous similar messages [ 5185.689433] LustreError: 186088:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5185.697392] LustreError: 186088:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5185.824280] Lustre: server umount lustre-MDT0000 complete [ 5188.073094] LustreError: 183430:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5188.099085] LustreError: 183430:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5189.735127] LustreError: 183411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250211 with bad export cookie 2144898930294488632 [ 5189.740197] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.754045] LustreError: 183411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5189.909244] LustreError: 186290:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5189.939804] LustreError: 186290:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5190.144800] Lustre: server umount lustre-MDT0001 complete [ 5201.284124] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5204.051094] Key type lgssc unregistered [ 5204.360739] LNet: 186821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5204.381503] LNetError: 186821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5204.396808] LNet: Removed LNI 192.168.203.160@tcp [ 5205.034680] Key type .llcrypt unregistered [ 5205.037227] Key type ._llcrypt unregistered [ 5225.145487] Key type ._llcrypt registered [ 5225.147045] Key type .llcrypt registered [ 5225.257443] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5226.015826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5226.098395] alg: No test for adler32 (adler32-zlib) [ 5227.224676] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5227.449991] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5229.120279] Key type lgssc registered [ 5229.980793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5236.917254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5236.941644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.463933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.311675] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5247.986050] LustreError: 188193:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.005714] LustreError: 188193:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5251.140742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5251.668792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5255.220981] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5260.543954] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5264.784783] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5271.433947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5271.702986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.211643] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5281.105477] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5282.986799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5290.804758] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5300.135628] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5306.352838] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5312.486349] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5324.268737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.275878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5324.276753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5329.380648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5329.386514] Lustre: Skipped 2 previous similar messages [ 5329.526288] LustreError: 191933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5329.652272] Lustre: server umount lustre-OST0000 complete [ 5333.473276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5333.484745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.504420] Lustre: Skipped 1 previous similar message [ 5333.512240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.501867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.617419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5339.623948] Lustre: Skipped 3 previous similar messages [ 5344.740499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.746202] Lustre: Skipped 1 previous similar message [ 5347.296227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5347.364283] LustreError: 192135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5347.377841] LustreError: 192135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5347.509170] Lustre: server umount lustre-MDT0000 complete [ 5349.858525] LustreError: 188921:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.865665] LustreError: 188921:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5350.418228] LustreError: 190545:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250371 with bad export cookie 16652939660434512825 [ 5350.424238] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.612564] LustreError: 192337:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5350.619125] LustreError: 192337:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5350.815971] Lustre: server umount lustre-MDT0001 complete [ 5361.328239] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5363.739573] Key type lgssc unregistered [ 5364.004098] LNet: 192869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5364.020760] LNetError: 192869:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5364.036116] LNet: Removed LNI 192.168.203.160@tcp [ 5364.607482] Key type .llcrypt unregistered [ 5364.609546] Key type ._llcrypt unregistered [ 5385.100628] Key type ._llcrypt registered [ 5385.102621] Key type .llcrypt registered [ 5385.219880] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5386.090230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5386.129319] alg: No test for adler32 (adler32-zlib) [ 5387.178869] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5387.368754] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5389.032178] Key type lgssc registered [ 5390.100099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5396.643866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5396.683275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.131134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.561893] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5407.719560] LustreError: 194242:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.726897] LustreError: 194242:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5408.532845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5408.802704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5411.327479] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5414.709823] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5416.907632] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5421.586577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5421.928345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.969639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5426.050952] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5430.650401] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5444.582288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.589555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.590431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5444.598502] Lustre: Skipped 1 previous similar message [ 5449.702967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.713210] Lustre: Skipped 2 previous similar messages [ 5449.785668] LustreError: 196639:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5450.089142] Lustre: server umount lustre-OST0000 complete [ 5454.818227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5454.818532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.825916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5454.834732] Lustre: Skipped 1 previous similar message [ 5459.941933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5459.951435] Lustre: Skipped 2 previous similar messages [ 5465.060273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5465.068932] Lustre: Skipped 1 previous similar message [ 5467.104203] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5467.199406] LustreError: 196841:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5467.204935] LustreError: 196841:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5467.339263] Lustre: server umount lustre-MDT0000 complete [ 5470.181747] LustreError: 196405:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.199294] LustreError: 196405:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5470.328477] LustreError: 194222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250491 with bad export cookie 4456110350434546280 [ 5470.331132] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.341253] LustreError: 194222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5470.494405] LustreError: 197044:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5470.497804] LustreError: 197044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5470.703478] Lustre: server umount lustre-MDT0001 complete [ 5480.667280] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5483.504596] Key type lgssc unregistered [ 5483.762549] LNet: 197577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5483.769728] LNetError: 197577:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5483.782058] LNet: Removed LNI 192.168.203.160@tcp [ 5484.603269] Key type .llcrypt unregistered [ 5484.604718] Key type ._llcrypt unregistered [ 5501.543505] Key type ._llcrypt registered [ 5501.545724] Key type .llcrypt registered [ 5501.650296] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5502.757246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.790613] alg: No test for adler32 (adler32-zlib) [ 5503.938971] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5504.229615] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5505.928209] Key type lgssc registered [ 5507.356705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5515.185584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5515.203423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.797525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5520.875214] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5527.012825] LustreError: 198824:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5527.025297] LustreError: 198824:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5529.116328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5529.701671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5533.365814] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5538.107409] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5541.509927] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5547.597252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5547.888378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.649626] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5557.259123] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5562.229503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5575.649681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5575.653421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.663573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.697498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5577.701195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5577.704856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.717585] Lustre: Skipped 1 previous similar message [ 5579.203144] LustreError: 201225:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5579.323096] Lustre: server umount lustre-OST0000 complete [ 5582.820286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.827110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.837569] Lustre: Skipped 1 previous similar message [ 5582.855384] Lustre: Skipped 1 previous similar message [ 5587.938413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5587.945469] Lustre: Skipped 1 previous similar message [ 5588.668783] LustreError: 201426:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5588.672490] LustreError: 201426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5588.740187] Lustre: server umount lustre-MDT0000 complete [ 5591.373297] LustreError: 198804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250612 with bad export cookie 9193020001023763828 [ 5591.374072] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5591.383557] LustreError: 198804:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5591.504133] LustreError: 201629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5591.513314] LustreError: 201629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5591.744312] Lustre: server umount lustre-MDT0001 complete [ 5601.385494] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5603.413322] Key type lgssc unregistered [ 5603.628475] LNet: 202160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5603.636608] LNetError: 202160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5603.658663] LNet: Removed LNI 192.168.203.160@tcp [ 5604.223575] Key type .llcrypt unregistered [ 5604.225865] Key type ._llcrypt unregistered [ 5617.476563] Key type ._llcrypt registered [ 5617.478223] Key type .llcrypt registered [ 5617.567194] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5618.456069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5618.485434] alg: No test for adler32 (adler32-zlib) [ 5619.461541] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5619.612292] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5621.242157] Key type lgssc registered [ 5622.182951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5627.906545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5627.931517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.358963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.490621] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5638.626945] LustreError: 203383:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5638.637364] LustreError: 203383:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5638.825834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5639.150752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5641.986667] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5646.269438] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5649.029995] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5654.172242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5654.514112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.388496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5659.068659] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5663.818572] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5672.756079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5672.874637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5673.141652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5678.612162] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5683.789120] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5685.812120] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5697.180843] Lustre: Permanently deactivating lustre-OST0001 [ 5697.201392] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5706.302564] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5715.812914] Lustre: Permanently reactivating lustre-OST0001 [ 5715.838260] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5715.849768] Lustre: Skipped 2 previous similar messages [ 5722.701643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.716053] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5722.726858] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5722.742801] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5722.764300] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5728.228105] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5728.232333] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5728.239753] Lustre: Skipped 1 previous similar message [ 5728.246691] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5729.253202] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5729.270593] Lustre: Skipped 2 previous similar messages [ 5731.514113] LustreError: 206880:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5731.585082] Lustre: server umount lustre-OST0001 complete [ 5734.374273] LustreError: 204984:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.388314] LustreError: 204984:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5735.396735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5735.412961] LustreError: Skipped 1 previous similar message [ 5735.415095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.440108] Lustre: Skipped 1 previous similar message [ 5735.451223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.442953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5737.446188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.459767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5739.491486] LustreError: 205516:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.506662] LustreError: 205516:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5744.611492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5744.614434] LustreError: 204985:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.625141] Lustre: Skipped 3 previous similar messages [ 5744.631147] LustreError: 204985:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5749.216133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5749.281198] LustreError: 207082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5749.283769] LustreError: 207082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5749.470183] Lustre: server umount lustre-OST0000 complete [ 5754.850218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.860685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.867076] Lustre: Skipped 1 previous similar message [ 5759.146308] LustreError: 207284:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5759.151112] LustreError: 207284:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5759.235801] Lustre: server umount lustre-MDT0000 complete [ 5759.973535] LustreError: 204098:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.989472] LustreError: 204098:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5762.339078] LustreError: 206559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250783 with bad export cookie 12147619456272348109 [ 5762.345291] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.349676] LustreError: 206559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5762.511186] LustreError: 207485:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5762.535698] LustreError: 207485:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5762.788320] Lustre: server umount lustre-MDT0001 complete [ 5772.488586] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 5774.937863] Key type lgssc unregistered [ 5775.239060] LNet: 208016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5775.248190] LNetError: 208016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5775.265478] LNet: Removed LNI 192.168.203.160@tcp [ 5775.987051] Key type .llcrypt unregistered [ 5775.989402] Key type ._llcrypt unregistered [ 5804.349091] Key type ._llcrypt registered [ 5804.354806] Key type .llcrypt registered [ 5804.488306] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 5817.268804] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 5818.259436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5818.286524] alg: No test for adler32 (adler32-zlib) [ 5819.327856] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5819.583977] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 5821.216163] Key type lgssc registered [ 5822.296629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5830.315737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5836.189664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5841.524373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5848.069333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.073750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.118765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5854.128041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.322636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5855.354407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.436872] Lustre: lustre-MDT0000: new disk, initializing [ 5855.565531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.589167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5859.073304] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5868.335742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.412719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5868.488469] Lustre: 211953:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5868.551153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5868.560135] Lustre: Skipped 1 previous similar message [ 5868.726097] Lustre: lustre-MDT0001: new disk, initializing [ 5868.877186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5868.936857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5868.952168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5869.058690] Lustre: 212750:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 5869.079429] Lustre: 212750:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 5869.091413] Lustre: 212750:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 5869.104252] Lustre: 212750:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 5869.111478] Lustre: 212750:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 5869.117465] Lustre: 212750:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 5873.268689] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5880.999828] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5885.398437] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5892.015674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5892.083562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5892.310901] Lustre: lustre-OST0000: new disk, initializing [ 5892.314801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5892.410227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.793739] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5900.375179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5900.389942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5900.496175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5904.290491] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5908.182931] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5908.376535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5911.568578] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5911.770911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5916.138085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.139244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5916.150648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5916.161482] Lustre: Skipped 1 previous similar message [ 5919.416224] LustreError: 214892:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5919.598401] Lustre: server umount lustre-OST0000 complete [ 5925.345846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5925.348123] LustreError: Skipped 1 previous similar message [ 5925.352637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.374557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.376718] Lustre: Skipped 1 previous similar message [ 5926.370169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5926.379154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5926.381949] Lustre: Skipped 1 previous similar message [ 5931.492406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5931.503188] Lustre: Skipped 1 previous similar message [ 5935.591673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5935.594394] Lustre: Skipped 1 previous similar message [ 5937.120124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5937.233585] LustreError: 215094:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5937.238252] LustreError: 215094:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5937.303955] Lustre: server umount lustre-MDT0000 complete [ 5940.310605] LustreError: 211946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250961 with bad export cookie 8685085286412032193 [ 5940.319608] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5940.457137] LustreError: 215297:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5940.460355] LustreError: 215297:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5940.656952] Lustre: server umount lustre-MDT0001 complete [ 5947.805176] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 16:22:47 (1761250967) [ 5953.286574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.802552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.262901] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5959.140552] LustreError: 215969:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.163837] LustreError: 215969:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5964.265447] LustreError: 215969:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.660795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5965.048492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5969.110765] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5974.601373] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5978.027678] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5983.304585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5983.565646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.169498] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 5993.288350] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6000.268332] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6002.457465] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6010.322643] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6019.771366] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6030.336851] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6046.348753] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6046.353446] Lustre: Skipped 1 previous similar message [ 6084.087108] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6084.094292] Lustre: Skipped 3 previous similar messages [ 6148.346939] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6148.352803] Lustre: Skipped 8 previous similar messages [ 6173.159834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.163823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.213620] Lustre: Skipped 1 previous similar message [ 6176.061136] LustreError: 219447:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6176.071392] LustreError: 219447:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6176.292826] Lustre: server umount lustre-OST0000 complete [ 6180.324348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6180.329901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.814504] LustreError: 219649:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6185.848268] LustreError: 219649:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6186.141081] Lustre: server umount lustre-MDT0000 complete [ 6188.514362] LustreError: 215969:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.768723] LustreError: 218337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251211 with bad export cookie 8685085286412033131 [ 6189.775994] LustreError: 218337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6189.770596] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.220966] Lustre: server umount lustre-MDT0001 complete [ 6198.694656] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6201.082460] Key type lgssc unregistered [ 6201.351092] LNet: 220382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6201.357977] LNetError: 220382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6201.379585] LNet: Removed LNI 192.168.203.160@tcp [ 6202.031624] Key type .llcrypt unregistered [ 6202.035085] Key type ._llcrypt unregistered [ 6223.366476] Key type ._llcrypt registered [ 6223.371649] Key type .llcrypt registered [ 6223.486735] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6224.443458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6224.477872] alg: No test for adler32 (adler32-zlib) [ 6225.526741] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 6225.701967] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6227.361589] Key type lgssc registered [ 6228.605633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6235.351785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6235.368575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.800228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.304416] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6247.398225] LustreError: 221756:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.405184] LustreError: 221756:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6248.642655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6248.964799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6251.968220] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6256.248477] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6259.852814] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6266.119529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6266.380179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.707713] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_default_debug -1 all [ 6276.519373] Lustre: DEBUG MARKER: oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6297.058821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.077643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6297.088204] Lustre: Skipped 1 previous similar message [ 6302.188014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6302.655565] LustreError: 224162:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6302.828170] Lustre: server umount lustre-OST0000 complete [ 6307.304463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6307.307526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6307.310866] Lustre: Skipped 2 previous similar messages [ 6307.319536] Lustre: Skipped 2 previous similar messages [ 6312.103179] LustreError: 224364:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6312.116906] LustreError: 224364:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6312.207818] Lustre: server umount lustre-MDT0000 complete [ 6312.418193] LustreError: 221755:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.430392] LustreError: 221755:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6315.390074] LustreError: 221736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251336 with bad export cookie 17275808470778481531 [ 6315.392166] LustreError: MGC192.168.203.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6315.408590] LustreError: 221736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.517321] LustreError: 224567:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6315.526148] LustreError: 224567:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6315.676432] Lustre: server umount lustre-MDT0001 complete [ 6323.034813] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6325.179932] Key type lgssc unregistered [ 6325.449553] LNet: 225098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6325.460294] LNetError: 225098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6325.473437] LNet: Removed LNI 192.168.203.160@tcp [ 6325.993155] Key type .llcrypt unregistered [ 6325.998474] Key type ._llcrypt unregistered [ 6341.727563] Key type ._llcrypt registered [ 6341.728991] Key type .llcrypt registered [ 6341.796946] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6341.985400] Key type .llcrypt unregistered [ 6341.987199] Key type ._llcrypt unregistered [ 6362.282421] Key type ._llcrypt registered [ 6362.287910] Key type .llcrypt registered [ 6362.357914] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing set_hostid [ 6374.246171] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing load_modules_local [ 6374.907311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6374.997967] alg: No test for adler32 (adler32-zlib) [ 6376.011562] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 6376.288247] LNet: Added LNI 192.168.203.160@tcp [8/256/0/180] [ 6377.992590] Key type lgssc registered [ 6378.997699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6386.469990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.857263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6398.963438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6405.988345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6407.721285] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6154 sec ========== 16:30:27 (1761251427) [ 6409.376587] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:30:29 (1761251429) === [ 6412.403442] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:30:32 (1761251432) === [ 6447.076371] Lustre: DEBUG MARKER: oleg360-server.virtnet: executing unload_modules_local [ 6449.712555] Key type lgssc unregistered [ 6450.052304] LNet: 231184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6450.059616] LNetError: 231184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6450.082551] LNet: Removed LNI 192.168.203.160@tcp [ 6450.767338] Key type .llcrypt unregistered [ 6450.768979] Key type ._llcrypt unregistered