[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 473934219 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002312] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007377] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.011045] Yama: becoming mindful. [ 0.012026] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.020743] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025003] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] *** VALIDATE tmpfs *** [ 0.030187] *** VALIDATE proc *** [ 0.031120] *** VALIDATE cgroup *** [ 0.031902] *** VALIDATE cgroup2 *** [ 0.033057] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036024] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040384] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.042156] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043611] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.044895] ... bit width: 48 [ 0.045011] ... generic registers: 4 [ 0.045965] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047011] ... fixed-purpose events: 3 [ 0.047954] ... event mask: 000000070000000f [ 0.048274] rcu: Hierarchical SRCU implementation. [ 0.050416] smp: Bringing up secondary CPUs ... [ 0.051513] x86: Booting SMP configuration: [ 0.052018] .... node #0, CPUs: #1 #2 #3 [ 0.055442] smp: Brought up 1 node, 4 CPUs [ 0.056982] smpboot: Max logical packages: 1 [ 0.057011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146248] node 0 deferred pages initialised in 87ms [ 0.149010] devtmpfs: initialized [ 0.150217] x86/mm: Memory block size: 128MB [ 0.152705] gcov: version magic: 0x41383552 [ 0.153684] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156060] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159465] pinctrl core: initialized pinctrl subsystem [ 0.161254] [ 0.161812] ************************************************************* [ 0.164010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166008] ** ** [ 0.168013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170012] ** ** [ 0.171007] ** This means that this kernel is built to expose internal ** [ 0.173010] ** IOMMU data structures, which may compromise security on ** [ 0.174008] ** your system. ** [ 0.175011] ** ** [ 0.177007] ** If you see this message and you are not debugging the ** [ 0.178006] ** kernel, report this immediately to your vendor! ** [ 0.179008] ** ** [ 0.181008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182006] ************************************************************* [ 0.183598] NET: Registered protocol family 16 [ 0.185298] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.189049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191086] cpuidle: using governor menu [ 0.192771] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194427] PCI: Using configuration type 1 for base access [ 0.197120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206051] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209064] cryptd: max_cpu_qlen set to 1000 [ 0.211720] ACPI: Added _OSI(Module Device) [ 0.213010] ACPI: Added _OSI(Processor Device) [ 0.214000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221404] ACPI: Interpreter enabled [ 0.223051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224011] ACPI: Using IOAPIC for interrupt routing [ 0.225125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236745] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243233] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247135] acpiphp: Slot [2] registered [ 0.248114] acpiphp: Slot [5] registered [ 0.249108] acpiphp: Slot [6] registered [ 0.250092] acpiphp: Slot [7] registered [ 0.252087] acpiphp: Slot [8] registered [ 0.253112] acpiphp: Slot [9] registered [ 0.254114] acpiphp: Slot [10] registered [ 0.255065] acpiphp: Slot [3] registered [ 0.256085] acpiphp: Slot [4] registered [ 0.257074] acpiphp: Slot [11] registered [ 0.259066] acpiphp: Slot [12] registered [ 0.260081] acpiphp: Slot [13] registered [ 0.261091] acpiphp: Slot [14] registered [ 0.262076] acpiphp: Slot [15] registered [ 0.264065] acpiphp: Slot [16] registered [ 0.265061] acpiphp: Slot [17] registered [ 0.266082] acpiphp: Slot [18] registered [ 0.268064] acpiphp: Slot [19] registered [ 0.269071] acpiphp: Slot [20] registered [ 0.271113] acpiphp: Slot [21] registered [ 0.272075] acpiphp: Slot [22] registered [ 0.273084] acpiphp: Slot [23] registered [ 0.274093] acpiphp: Slot [24] registered [ 0.276104] acpiphp: Slot [25] registered [ 0.277128] acpiphp: Slot [26] registered [ 0.278080] acpiphp: Slot [27] registered [ 0.279118] acpiphp: Slot [28] registered [ 0.281081] acpiphp: Slot [29] registered [ 0.282093] acpiphp: Slot [30] registered [ 0.283073] acpiphp: Slot [31] registered [ 0.285087] PCI host bridge to bus 0000:00 [ 0.286014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297149] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299031] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302086] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310610] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315084] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.320019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.322014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324602] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.326677] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.328036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.330539] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.343015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.353560] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.382015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.389959] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.403015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.408014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.420015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.433013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.438016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.456023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.467629] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.491013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.502017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.510598] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.517015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.522016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.533016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.544087] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.553032] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.568015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.592017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.604497] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.608279] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.611656] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.615334] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.622248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.629075] iommu: Default domain type: Passthrough [ 0.631463] SCSI subsystem initialized [ 0.632103] ACPI: bus type USB registered [ 0.633071] usbcore: registered new interface driver usbfs [ 0.635087] usbcore: registered new interface driver hub [ 0.637081] usbcore: registered new device driver usb [ 0.639245] pps_core: LinuxPPS API ver. 1 registered [ 0.640000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.642053] PTP clock support registered [ 0.644112] EDAC MC: Ver: 3.0.0 [ 0.646083] PCI: Using ACPI for IRQ routing [ 0.648098] NetLabel: Initializing [ 0.650009] NetLabel: domain hash size = 128 [ 0.651007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.652111] NetLabel: unlabeled traffic allowed by default [ 0.654106] vgaarb: loaded [ 0.656202] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.657009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.663000] clocksource: Switched to clocksource kvm-clock [ 0.779213] VFS: Disk quotas dquot_6.6.0 [ 0.780774] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782975] *** VALIDATE ramfs *** [ 0.783937] *** VALIDATE hugetlbfs *** [ 0.785186] pnp: PnP ACPI init [ 0.787239] pnp: PnP ACPI: found 6 devices [ 0.811954] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.814779] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.816745] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.818676] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.820695] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.822702] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.825131] NET: Registered protocol family 2 [ 0.827306] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.831771] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.834724] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.840202] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.843927] TCP: Hash tables configured (established 65536 bind 65536) [ 0.846996] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.850457] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853502] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856939] NET: Registered protocol family 1 [ 0.860480] RPC: Registered named UNIX socket transport module. [ 0.862920] RPC: Registered udp transport module. [ 0.864650] RPC: Registered tcp transport module. [ 0.867067] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.869744] NET: Registered protocol family 44 [ 0.871549] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.873482] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.876497] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.879737] PCI: CLS 0 bytes, default 64 [ 0.882108] Unpacking initramfs... [ 2.332653] debug: unmapping init [mem 0xffff99d33cc54000-0xffff99d33ffbffff] [ 2.336054] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.337864] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.339929] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.826386] Initialise system trusted keyrings [ 2.828281] Key type blacklist registered [ 2.830295] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.840784] zbud: loaded [ 2.844414] *** VALIDATE nfs *** [ 2.845889] *** VALIDATE nfs4 *** [ 2.849061] pstore: using deflate compression [ 2.853410] Platform Keyring initialized [ 2.951799] NET: Registered protocol family 38 [ 2.954060] Key type asymmetric registered [ 2.955505] Asymmetric key parser 'x509' registered [ 2.956950] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.959453] io scheduler mq-deadline registered [ 2.961313] io scheduler kyber registered [ 2.962745] io scheduler bfq registered [ 2.964201] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.966773] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.969108] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.971719] ACPI: Power Button [PWRF] [ 2.977180] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.984277] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.996369] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.004536] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.019643] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.048169] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.078430] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.083215] Non-volatile memory driver v1.3 [ 3.084865] Linux agpgart interface v0.103 [ 3.114148] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.116392] vda: detected capacity change from 0 to 68624384 [ 3.134611] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.138263] vdb: detected capacity change from 0 to 1073741824 [ 3.157153] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.159364] vdc: detected capacity change from 0 to 2621440000 [ 3.173033] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.175103] vdd: detected capacity change from 0 to 2621440000 [ 3.187720] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.190632] vde: detected capacity change from 0 to 4294967296 [ 3.203384] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.206303] vdf: detected capacity change from 0 to 4294967296 [ 3.212355] libphy: Fixed MDIO Bus: probed [ 3.220766] usbcore: registered new interface driver usbserial_generic [ 3.223186] usbserial: USB Serial support registered for generic [ 3.225605] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.230195] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.231848] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.234647] mousedev: PS/2 mouse device common for all mice [ 3.238719] rtc_cmos 00:05: RTC can wake from S4 [ 3.243032] rtc_cmos 00:05: registered as rtc0 [ 3.245789] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.246293] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.252164] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.253470] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.269667] intel_pstate: CPU model not supported [ 3.273487] hid: raw HID events driver (C) Jiri Kosina [ 3.276237] usbcore: registered new interface driver usbhid [ 3.278777] usbhid: USB HID core driver [ 3.281108] drop_monitor: Initializing network drop monitor service [ 3.284134] Initializing XFRM netlink socket [ 3.286207] NET: Registered protocol family 10 [ 3.289249] Segment Routing with IPv6 [ 3.291065] NET: Registered protocol family 17 [ 3.293700] mpls_gso: MPLS GSO support [ 3.299219] RAS: Correctable Errors collector initialized. [ 3.301789] AVX version of gcm_enc/dec engaged. [ 3.303339] AES CTR mode by8 optimization enabled [ 3.378712] sched_clock: Marking stable (3378692106, 0)->(4181638177, -802946071) [ 3.382437] registered taskstats version 1 [ 3.384749] Loading compiled-in X.509 certificates [ 3.386946] zswap: loaded using pool lzo/zbud [ 3.411610] Key type big_key registered [ 3.422573] Key type encrypted registered [ 3.423741] ima: No TPM chip found, activating TPM-bypass! [ 3.424899] ima: Allocated hash algorithm: sha1 [ 3.426089] ima: No architecture policies found [ 3.427640] evm: Initialising EVM extended attributes: [ 3.428829] evm: security.selinux [ 3.429601] evm: security.ima [ 3.430273] evm: security.capability [ 3.430974] evm: HMAC attrs: 0x1 [ 3.432416] rtc_cmos 00:05: setting system clock to 2026-01-02 00:54:24 UTC (1767315264) [ 3.436709] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.439380] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 3.447111] Write protecting the kernel read-only data: 28672k [ 3.450047] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.452376] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 3.483651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.491349] systemd[1]: Detected virtualization kvm. [ 3.493495] systemd[1]: Detected architecture x86-64. [ 3.495703] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.522693] systemd[1]: No hostname configured. [ 3.524399] systemd[1]: Set hostname to . [ 3.526432] random: systemd: uninitialized urandom read (16 bytes read) [ 3.528597] systemd[1]: Initializing machine ID from random generator. [ 3.563666] random: ln: uninitialized urandom read (6 bytes read) [ 3.649379] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651436] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.654787] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.658363] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.262438] device-mapper: uevent: version 1.0.3 [ 4.264342] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.017475] virtio_net virtio0 ens2: renamed from eth0 [ 5.018988] random: fast init done [ 5.186595] scsi host0: ata_piix [ 5.263665] scsi host1: ata_piix [ 5.265331] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.267786] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.705913] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.935078] random: crng init done [ 9.936493] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.480801] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.809393] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.079542] SELinux: Disabled at runtime. [ 12.140969] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.151108] systemd[1]: Detected virtualization kvm. [ 12.153197] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.651390] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.654438] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.659689] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.666227] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.669980] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.677937] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.686485] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK [[ 12.806302] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.334990] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.672362] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.735154] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.859952] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.874382] EDAC sbridge: Ver: 1.1.2 [ 15.453092] Key type dns_resolver registered [ 15.769395] NFS: Registering the id_resolver key type [ 15.771655] Key type id_resolver registered [ 15.773583] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg347-server login: [ 47.291461] libcfs: loading out-of-tree module taints kernel. [ 47.327547] Key type ._llcrypt registered [ 47.329777] Key type .llcrypt registered [ 47.416737] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 119.913814] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 121.725394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.756041] alg: No test for adler32 (adler32-zlib) [ 123.304928] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 124.329668] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 126.159164] Key type lgssc registered [ 127.851453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 155.450247] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.384675] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.600230] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.823778] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.163775] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 253.591170] hrtimer: interrupt took 2492375 ns [ 270.108890] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 293.462790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 293.516399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 293.534836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 294.881679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 294.940938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 295.098371] Lustre: lustre-MDT0000: new disk, initializing [ 295.238848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.267323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 300.401423] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 308.316571] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 325.278642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 325.388528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 325.504949] Lustre: 14431:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 325.578374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 325.584193] Lustre: Skipped 1 previous similar message [ 325.729112] Lustre: lustre-MDT0001: new disk, initializing [ 325.904386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 325.958347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 325.982456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 330.985814] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 338.716226] Lustre: 16494:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 355.269109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 355.433367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 355.811508] Lustre: lustre-OST0000: new disk, initializing [ 355.816595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 355.911289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.255587] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 366.151313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 366.167779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 366.345983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.523528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.665744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 394.879490] Lustre: lustre-OST0001: new disk, initializing [ 394.890719] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 395.002876] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 403.044072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 403.060626] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 403.139967] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 403.949383] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 417.050296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 423.417065] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 436.188535] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing check_logdir /tmp/testlogs/ [ 442.435083] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing yml_node [ 451.000301] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 453.802089] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 457.072644] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 458.988130] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 1 20:01:59 EST 2026 [ 483.636157] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 485.200149] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 505.312112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.316898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.339527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 505.825106] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.851641] Lustre: Skipped 2 previous similar messages [ 505.884708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 505.891582] Lustre: Skipped 2 previous similar messages [ 507.857450] LustreError: 21902:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 508.316929] Lustre: server umount lustre-MDT0000 complete [ 516.070902] LustreError: 20452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.109304] LustreError: 20452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 517.410110] LustreError: 14424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315778 with bad export cookie 8459070196856505306 [ 517.419208] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.423984] LustreError: 14424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 521.183825] LustreError: 20452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.184981] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.207397] LustreError: 20452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 521.248656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 522.213114] LustreError: 14439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.247843] LustreError: 14439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 523.768936] LustreError: 22353:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 523.782549] LustreError: 22353:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 524.021258] Lustre: server umount lustre-MDT0001 complete [ 535.719612] LustreError: 22948:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 535.724561] LustreError: 22948:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 535.977607] Lustre: server umount lustre-OST0000 complete [ 544.439309] LustreError: 23399:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 544.447625] LustreError: 23399:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 544.561255] Lustre: server umount lustre-OST0001 complete [ 559.248871] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 627.353600] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 649.301227] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 662.718763] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 673.981198] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 685.486788] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 703.362893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 703.486092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.030852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 704.102844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 704.223625] Lustre: lustre-MDT0000: new disk, initializing [ 704.301375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 704.311895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 708.750582] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 726.426122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 726.515767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 726.652885] Lustre: 30610:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 726.710174] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 726.713826] Lustre: Skipped 1 previous similar message [ 726.778928] Lustre: lustre-MDT0001: new disk, initializing [ 726.899963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.957814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 726.972805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 731.346161] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 739.667795] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.663741] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 757.617742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 757.698453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 758.055920] Lustre: lustre-OST0000: new disk, initializing [ 758.063921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 758.197580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.101275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 760.110476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 760.329566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 765.260236] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 773.135362] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 778.949708] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 779.336281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 785.418884] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 785.835731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 791.022726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.033732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.041483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 791.063199] Lustre: Skipped 2 previous similar messages [ 791.084136] Lustre: Skipped 4 previous similar messages [ 794.361372] LustreError: 34243:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 794.364064] LustreError: 34243:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 794.461950] Lustre: server umount lustre-OST0000 complete [ 801.249800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.262045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 801.279744] Lustre: Skipped 1 previous similar message [ 806.691726] LustreError: 34492:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 806.706227] LustreError: 34492:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 806.889111] Lustre: server umount lustre-MDT0000 complete [ 811.489729] LustreError: 30618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.520198] LustreError: 30618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 812.510513] LustreError: 30603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316073 with bad export cookie 8459070196856506860 [ 812.524838] LustreError: 30603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 812.529345] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 813.330889] Lustre: server umount lustre-MDT0001 complete [ 817.739552] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:07:58 (1767316078) [ 829.970280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.847292] LustreError: 35460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.955131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.013887] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 840.177917] LustreError: 35459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.207782] LustreError: 35459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 852.856434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 858.653346] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 866.398516] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 870.988894] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 883.413499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 891.460854] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 897.274744] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 913.888541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 913.902056] LustreError: Skipped 1 previous similar message [ 913.904198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.925554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 913.934844] Lustre: Skipped 3 previous similar messages [ 915.946765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 917.127615] LustreError: 38114:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 917.139583] LustreError: 38114:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 917.393940] Lustre: server umount lustre-OST0000 complete [ 925.155344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 929.508432] Lustre: server umount lustre-MDT0000 complete [ 931.303949] LustreError: 36365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.325397] LustreError: 36365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 934.355352] LustreError: 35440:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316195 with bad export cookie 8459070196856507742 [ 934.368398] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 934.994994] Lustre: server umount lustre-MDT0001 complete [ 948.753857] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 951.510578] Key type lgssc unregistered [ 951.862737] LNet: 39290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.877800] LNetError: 39290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.903564] LNet: Removed LNI 192.168.203.147@tcp [ 953.043148] Key type .llcrypt unregistered [ 953.044991] Key type ._llcrypt unregistered [ 1033.985542] Key type ._llcrypt registered [ 1033.987852] Key type .llcrypt registered [ 1034.108967] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1035.729157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1036.045739] alg: No test for adler32 (adler32-zlib) [ 1037.159677] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1037.544678] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1039.343147] Key type lgssc registered [ 1040.780744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.003831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1049.019937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1050.506787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.714958] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1060.837226] LustreError: 43748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.865566] LustreError: 43748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1065.708561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1066.003212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.668695] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1076.767506] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1080.149825] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1090.499462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1090.879063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.973264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1097.097330] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1102.431770] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1123.843135] LustreError: 47015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1124.109022] Lustre: server umount lustre-OST0000 complete [ 1130.983804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.989215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1131.008099] Lustre: Skipped 1 previous similar message [ 1131.015531] Lustre: Skipped 1 previous similar message [ 1134.049633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.070414] Lustre: Skipped 1 previous similar message [ 1139.168866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.176194] Lustre: Skipped 1 previous similar message [ 1141.215706] Lustre: 43060:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316386/real 1767316386] req@000000008c9c8b23 x1853165454521216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767316402 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1141.215930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.250943] Lustre: 43060:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1141.262511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.268728] Lustre: Skipped 1 previous similar message [ 1143.265291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1143.384455] LustreError: 47264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1143.392058] LustreError: 47264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1143.503395] Lustre: server umount lustre-MDT0000 complete [ 1146.336835] LustreError: 44913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.346130] LustreError: 44913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1147.167129] Lustre: 43059:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316392/real 1767316392] req@00000000736b0bb2 x1853165454523136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767316408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1148.344699] LustreError: 43728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316409 with bad export cookie 10196515324056329878 [ 1148.349395] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.355076] LustreError: 43728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1148.485910] LustreError: 47515:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1148.489659] LustreError: 47515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1148.723511] Lustre: server umount lustre-MDT0001 complete [ 1163.191948] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1166.243678] Key type lgssc unregistered [ 1166.539677] LNet: 48191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.545398] LNetError: 48191:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.558947] LNet: Removed LNI 192.168.203.147@tcp [ 1167.291176] Key type .llcrypt unregistered [ 1167.293564] Key type ._llcrypt unregistered [ 1252.450990] Key type ._llcrypt registered [ 1252.453844] Key type .llcrypt registered [ 1252.584780] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1253.843726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1253.876183] alg: No test for adler32 (adler32-zlib) [ 1255.240594] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1255.464165] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1257.145087] Key type lgssc registered [ 1258.530483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.188781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1267.215695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.772807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.196447] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1277.945660] LustreError: 52646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.965283] LustreError: 52646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1283.048271] LustreError: 52646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.166248] LustreError: 52646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.272926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1288.681088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.105849] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1299.756735] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.366647] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1327.098644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1327.488932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.632828] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1338.737058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1340.189950] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1349.097437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1349.103921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.110442] LustreError: Skipped 1 previous similar message [ 1349.111607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1349.148917] Lustre: Skipped 1 previous similar message [ 1354.110113] LustreError: 55820:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1354.222752] LustreError: 55683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.341576] Lustre: server umount lustre-OST0000 complete [ 1360.352835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.372875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.404223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.419641] Lustre: Skipped 1 previous similar message [ 1364.454230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.474340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1365.672252] LustreError: 56069:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1365.678437] LustreError: 56069:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1365.882427] Lustre: server umount lustre-MDT0000 complete [ 1369.568524] LustreError: 53820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.604825] LustreError: 53820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1371.195956] LustreError: 52624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316632 with bad export cookie 13430693870790824283 [ 1371.212390] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1371.402849] LustreError: 56320:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1371.411153] LustreError: 56320:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1371.867782] Lustre: server umount lustre-MDT0001 complete [ 1387.816921] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1390.626934] Key type lgssc unregistered [ 1390.969042] LNet: 56998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.006204] LNetError: 56998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.024593] LNet: Removed LNI 192.168.203.147@tcp [ 1391.831439] Key type .llcrypt unregistered [ 1391.835429] Key type ._llcrypt unregistered [ 1473.915909] Key type ._llcrypt registered [ 1473.918411] Key type .llcrypt registered [ 1474.024364] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1476.370036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.430994] alg: No test for adler32 (adler32-zlib) [ 1477.675061] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1477.998692] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1479.728394] Key type lgssc registered [ 1481.413372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1489.707330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1489.719351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.325045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.075416] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1501.673334] LustreError: 61468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.692314] LustreError: 61468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1506.793298] LustreError: 61468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.066352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1509.647471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.728986] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1522.765943] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1527.567625] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1537.191308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1537.615225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.950779] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1550.532584] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1561.156098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1571.318890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.324191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1571.324605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1571.346669] Lustre: Skipped 1 previous similar message [ 1571.382692] Lustre: Skipped 1 previous similar message [ 1576.134807] LustreError: 64123:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1576.366518] Lustre: server umount lustre-OST0000 complete [ 1581.225097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.236275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.240205] Lustre: Skipped 1 previous similar message [ 1581.260454] Lustre: Skipped 1 previous similar message [ 1586.658088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.667911] Lustre: Skipped 1 previous similar message [ 1587.452413] LustreError: 64374:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1587.467273] LustreError: 64374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1587.600643] Lustre: server umount lustre-MDT0000 complete [ 1591.777854] LustreError: 62621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.799810] LustreError: 62621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1592.439067] LustreError: 64124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316853 with bad export cookie 14384492912137780343 [ 1592.451541] LustreError: 64124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1592.451586] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1592.640196] LustreError: 64624:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1592.644579] LustreError: 64624:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1592.823990] Lustre: server umount lustre-MDT0001 complete [ 1606.654601] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1609.383226] Key type lgssc unregistered [ 1609.690448] LNet: 65299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.702979] LNetError: 65299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.732323] LNet: Removed LNI 192.168.203.147@tcp [ 1610.629109] Key type .llcrypt unregistered [ 1610.635516] Key type ._llcrypt unregistered [ 1690.400542] Key type ._llcrypt registered [ 1690.404474] Key type .llcrypt registered [ 1690.490765] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1691.345500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.424960] alg: No test for adler32 (adler32-zlib) [ 1692.517129] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1692.784353] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1694.439148] Key type lgssc registered [ 1695.930110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.381861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.419444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.092469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.161599] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1715.177805] LustreError: 69755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.223623] LustreError: 69755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1720.292378] LustreError: 69755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.538808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1724.088226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1729.173080] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1735.764175] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1740.321785] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1751.693900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.031678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.169642] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1766.258615] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1775.620683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1784.803560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1784.814458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.842458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.828900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1785.832042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.842899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.895029] Lustre: Skipped 1 previous similar message [ 1787.580201] Lustre: lustre-OST0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 1788.607104] LustreError: 72409:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1788.878421] Lustre: server umount lustre-OST0000 complete [ 1811.437940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1811.462822] Lustre: Skipped 1 previous similar message [ 1811.467057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1817.571267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1817.584653] Lustre: Skipped 4 previous similar messages [ 1817.754982] LustreError: 72835:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1817.765653] LustreError: 72835:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1818.142762] Lustre: server umount lustre-MDT0000 complete [ 1822.703172] LustreError: 70661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1822.739322] LustreError: 70661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1824.474929] LustreError: 72836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317085 with bad export cookie 14786701082307261845 [ 1824.482348] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1824.486843] LustreError: 72836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1830.762565] LustreError: 73086:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1830.767127] LustreError: 73086:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1830.946596] Lustre: server umount lustre-MDT0001 complete [ 1844.797423] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 1847.038267] Key type lgssc unregistered [ 1847.330025] LNet: 73763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.337426] LNetError: 73763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.357678] LNet: Removed LNI 192.168.203.147@tcp [ 1848.437190] Key type .llcrypt unregistered [ 1848.444146] Key type ._llcrypt unregistered [ 1936.743836] Key type ._llcrypt registered [ 1936.749188] Key type .llcrypt registered [ 1936.949655] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 1938.299731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.413733] alg: No test for adler32 (adler32-zlib) [ 1939.738577] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 1940.102472] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 1941.839185] Key type lgssc registered [ 1943.359623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.954074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1952.008700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.816840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.797089] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1962.979786] LustreError: 78220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.998128] LustreError: 78220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1968.101912] LustreError: 78220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1972.312822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1972.766297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1977.934677] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 1984.397855] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1988.617118] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.016801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2000.476262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.632359] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2014.198737] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2022.960395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2033.127607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.129297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.153571] Lustre: Skipped 1 previous similar message [ 2035.386765] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.47@tcp (stopping) [ 2035.390381] Lustre: Skipped 2 previous similar messages [ 2036.778297] LustreError: 80920:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2036.906479] Lustre: server umount lustre-MDT0000 complete [ 2038.242359] LustreError: 78216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.262081] LustreError: 78216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2040.474712] LustreError: 78215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.713187] LustreError: 80922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317302 with bad export cookie 4924028008365866565 [ 2041.714978] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.724967] LustreError: 80922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2043.361230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.380024] Lustre: Skipped 1 previous similar message [ 2043.391086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2045.583711] LustreError: 78217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.608294] LustreError: 78217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2045.612296] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [ 2050.711650] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.47@tcp (stopping) [ 2050.722478] Lustre: Skipped 1 previous similar message [ 2055.798187] LustreError: 78217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.814965] LustreError: 78217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2056.161040] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2056.274516] LustreError: 81172:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2056.283670] LustreError: 81172:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2056.632057] Lustre: server umount lustre-MDT0001 complete [ 2071.529208] LustreError: 81397:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2071.540206] LustreError: 81397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2071.845528] Lustre: server umount lustre-OST0000 complete [ 2093.644269] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2096.893330] Key type lgssc unregistered [ 2097.257513] LNet: 82474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.264455] LNetError: 82474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.280061] LNet: Removed LNI 192.168.203.147@tcp [ 2099.373326] Key type .llcrypt unregistered [ 2099.379082] Key type ._llcrypt unregistered [ 2183.367475] Key type ._llcrypt registered [ 2183.374605] Key type .llcrypt registered [ 2183.516704] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2184.699758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.763864] alg: No test for adler32 (adler32-zlib) [ 2185.801761] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 2186.015064] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2187.680217] Key type lgssc registered [ 2188.937836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.831856] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2198.863951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2214.559293] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2219.679199] LustreError: 86901:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 2246.303227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.000741] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2257.674266] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2428.610949] LustreError: 87755:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2428.771865] Lustre: server umount lustre-OST0000 complete [ 2449.345936] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2451.386953] Key type lgssc unregistered [ 2451.627557] LNet: 88832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.633523] LNetError: 88832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.650176] LNet: Removed LNI 192.168.203.147@tcp [ 2452.465621] Key type .llcrypt unregistered [ 2452.467597] Key type ._llcrypt unregistered [ 2534.378974] Key type ._llcrypt registered [ 2534.385035] Key type .llcrypt registered [ 2534.555981] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2535.917440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2536.009239] alg: No test for adler32 (adler32-zlib) [ 2537.201385] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 2537.535615] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2539.242716] Key type lgssc registered [ 2540.209418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2548.110495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2548.132791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.748426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.537349] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2558.952129] LustreError: 93431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.974704] LustreError: 93431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2564.068154] LustreError: 93431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.234051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2567.826205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.408911] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2578.159451] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2581.970472] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2592.041256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2592.514634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.707477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2599.198389] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2604.735879] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2609.124074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2609.129642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.140316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.144614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2610.155882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.157954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.196918] Lustre: Skipped 1 previous similar message [ 2613.183073] LustreError: 96062:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2613.273915] Lustre: server umount lustre-OST0000 complete [ 2618.849099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.860674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.886146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.101182] LustreError: 96313:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2624.116300] LustreError: 96313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2624.296165] Lustre: server umount lustre-MDT0000 complete [ 2625.504176] LustreError: 94336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.524435] LustreError: 94336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2628.876390] LustreError: 93410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317889 with bad export cookie 10865524103283388603 [ 2628.884501] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.026641] LustreError: 96563:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2629.030077] LustreError: 96563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2629.330155] Lustre: server umount lustre-MDT0001 complete [ 2642.419741] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2645.649339] Key type lgssc unregistered [ 2645.988481] LNet: 97240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.995509] LNetError: 97240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.014833] LNet: Removed LNI 192.168.203.147@tcp [ 2647.979033] Key type .llcrypt unregistered [ 2647.983104] Key type ._llcrypt unregistered [ 2733.799424] Key type ._llcrypt registered [ 2733.801475] Key type .llcrypt registered [ 2733.901875] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 2735.111364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2735.213089] alg: No test for adler32 (adler32-zlib) [ 2736.337853] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 2736.583955] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 2738.240671] Key type lgssc registered [ 2739.609670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.563371] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2748.590208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2764.192424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2769.312292] LustreError: 101666:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 2796.002523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.720418] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2808.623912] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2819.622725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.227808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.302681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2825.118331] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2825.696309] LustreError: 101692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.730104] LustreError: 101692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2830.825049] LustreError: 101692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.841675] LustreError: 101692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2835.942796] LustreError: 102784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.965342] LustreError: 102784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2837.740651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2838.017444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2842.048663] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 2847.857220] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2851.532930] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2853.354571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2853.363696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.375968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.472607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.479570] Lustre: Skipped 2 previous similar messages [ 2859.449182] LustreError: 104330:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2859.639641] Lustre: server umount lustre-OST0000 complete [ 2884.065178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2884.072687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.094408] Lustre: Skipped 1 previous similar message [ 2884.104797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2888.162106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.167285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2888.200588] Lustre: Skipped 1 previous similar message [ 2889.522849] LustreError: 104758:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2889.528526] LustreError: 104758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2889.623532] Lustre: server umount lustre-MDT0000 complete [ 2893.282684] LustreError: 102784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.303825] LustreError: 102784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2894.740589] LustreError: 101675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318155 with bad export cookie 9886231892041145449 [ 2894.743267] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.751719] LustreError: 101675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2909.155393] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2909.255784] LustreError: 105012:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2909.258926] LustreError: 105012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2909.474920] Lustre: server umount lustre-MDT0001 complete [ 2922.527910] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 2925.023032] Key type lgssc unregistered [ 2925.363549] LNet: 105692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2925.376423] LNetError: 105692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2925.401744] LNet: Removed LNI 192.168.203.147@tcp [ 2926.288199] Key type .llcrypt unregistered [ 2926.290365] Key type ._llcrypt unregistered [ 3013.626628] Key type ._llcrypt registered [ 3013.632332] Key type .llcrypt registered [ 3013.731378] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3014.886070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3014.956800] alg: No test for adler32 (adler32-zlib) [ 3016.045429] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3016.221679] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3017.879671] Key type lgssc registered [ 3019.077428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3026.714170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3026.753675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.375341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.405548] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3038.691721] LustreError: 110156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3038.706985] LustreError: 110156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3043.809201] LustreError: 110156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.589564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3047.413200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3053.810428] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3061.035946] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3065.739970] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3076.755171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3077.060885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.605704] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3089.187831] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3098.553397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 3107.817181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3107.826634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.846503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.832517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.838785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.842084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.861130] Lustre: Skipped 1 previous similar message [ 3113.347215] LustreError: 112809:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3113.604129] Lustre: server umount lustre-OST0000 complete [ 3119.073174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.076109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.092324] Lustre: Skipped 1 previous similar message [ 3119.109808] Lustre: Skipped 1 previous similar message [ 3124.194548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.696284] LustreError: 113061:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3124.702831] LustreError: 113061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3124.980701] Lustre: server umount lustre-MDT0000 complete [ 3129.314635] LustreError: 112830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3129.345442] LustreError: 112830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3129.656671] LustreError: 113063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318390 with bad export cookie 12698791911895407517 [ 3129.662893] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.674989] LustreError: 113063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3129.934312] LustreError: 113312:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3129.941394] LustreError: 113312:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3130.305538] Lustre: server umount lustre-MDT0001 complete [ 3145.220124] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3148.258659] Key type lgssc unregistered [ 3148.597641] LNet: 113992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.608168] LNetError: 113992:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3149.669648] LNet: Removed LNI 192.168.203.147@tcp [ 3150.467243] Key type .llcrypt unregistered [ 3150.470641] Key type ._llcrypt unregistered [ 3236.579648] Key type ._llcrypt registered [ 3236.581882] Key type .llcrypt registered [ 3236.712789] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3237.834647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3238.009067] alg: No test for adler32 (adler32-zlib) [ 3239.063019] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3239.254643] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3240.951715] Key type lgssc registered [ 3242.384692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3251.246759] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:48:32 (1767318512) [ 3260.374966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3260.399349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.830922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.036674] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3271.143789] LustreError: 118884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.158542] LustreError: 118884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3276.257885] LustreError: 118884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.137287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.573807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3282.525265] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3289.221882] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3293.674626] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3305.539453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3306.231386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.257695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 3313.262570] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3319.512116] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3330.003604] Lustre: Failing over lustre-MDT0000 [ 3330.019625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 3330.049859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.089030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3330.408159] LustreError: 121519:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3331.052179] Lustre: server umount lustre-MDT0000 complete [ 3332.830591] LustreError: 118879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.119273] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.887751] LustreError: 118879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3337.924541] LustreError: 118879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3343.021765] LustreError: 118878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.059254] LustreError: 118878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3348.101786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.303508] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3348.470219] Lustre: *** cfs_fail_loc=135, val=0*** [ 3348.563467] LustreError: 122221:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 3348.567771] LustreError: 122221:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.147@tcp: cfg command failed: rc = -2 [ 3348.577474] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3348.577474] [ 3348.584566] LustreError: MGC192.168.203.147@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3348.592823] LustreError: 122209:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3348.598518] LustreError: 122209:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3348.606544] LustreError: 122209:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3348.612917] LustreError: 122209:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 3348.629454] Lustre: server umount lustre-MDT0000 complete [ 3348.635528] LustreError: 122209:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3354.261274] LustreError: 120542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.47@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.279138] LustreError: 120542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3361.065043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.264647] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.296510] LustreError: 118875:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3361.660165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3361.710177] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3364.527665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3366.871705] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3366.899827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3366.943623] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3367.004728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3375.088805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.092858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.116834] Lustre: Skipped 1 previous similar message [ 3375.134108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.128802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.134225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.483296] LustreError: 123224:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3379.500776] LustreError: 123224:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3379.706967] Lustre: server umount lustre-OST0000 complete [ 3387.366387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3387.371902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.395038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.412498] Lustre: Skipped 1 previous similar message [ 3387.423982] Lustre: Skipped 1 previous similar message [ 3392.484485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3392.516633] Lustre: Skipped 2 previous similar messages [ 3400.159506] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3400.353052] LustreError: 123474:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3400.363931] LustreError: 123474:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3400.520752] Lustre: server umount lustre-MDT0000 complete [ 3402.720489] LustreError: 118879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.756177] LustreError: 118879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3405.222217] LustreError: 119798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318666 with bad export cookie 4941714367678174982 [ 3405.230508] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.239807] LustreError: 119798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3405.486342] LustreError: 123726:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3405.494556] LustreError: 123726:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3405.690549] Lustre: server umount lustre-MDT0001 complete [ 3420.504181] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3423.497531] Key type lgssc unregistered [ 3423.929181] LNet: 124403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3423.938739] LNetError: 124403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3423.949991] LNet: Removed LNI 192.168.203.147@tcp [ 3424.859514] Key type .llcrypt unregistered [ 3424.861910] Key type ._llcrypt unregistered [ 3510.008228] Key type ._llcrypt registered [ 3510.015203] Key type .llcrypt registered [ 3510.172598] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3511.175692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3511.297296] alg: No test for adler32 (adler32-zlib) [ 3512.404949] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3512.821284] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3514.584682] Key type lgssc registered [ 3515.743377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.272715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3524.291076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.792471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.861837] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3535.329732] LustreError: 128858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.347851] LustreError: 128858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3540.451213] LustreError: 128858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.194421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3543.682203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3547.659823] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3553.203400] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3557.261162] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3569.337970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3569.988725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.050723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3576.539837] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3582.388477] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3592.900474] Lustre: Failing over lustre-MDT0000 [ 3593.160701] LustreError: 131494:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3593.275151] Lustre: server umount lustre-MDT0000 complete [ 3593.697330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.702669] LustreError: 128855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.722128] Lustre: Skipped 2 previous similar messages [ 3593.751111] LustreError: 128855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3598.820630] LustreError: 128854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.836547] LustreError: 128854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3603.950054] LustreError: 130075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.969858] LustreError: 130075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3608.233499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.378432] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3608.633811] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3608.803712] LustreError: 132196:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 3608.816318] LustreError: 132196:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.147@tcp: cfg command failed: rc = -9 [ 3608.828785] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3608.828785] [ 3608.881486] LustreError: MGC192.168.203.147@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3608.908079] LustreError: 132184:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3608.919989] LustreError: 132184:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 3608.933879] LustreError: 132184:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3608.948690] LustreError: 132184:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 3608.986810] Lustre: server umount lustre-MDT0000 complete [ 3608.990242] LustreError: 132184:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3614.179623] LustreError: 128859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.211895] LustreError: 128859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3621.564208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.796873] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.815201] LustreError: 128850:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3622.240032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.350440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3626.132172] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3627.189392] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3627.501412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3627.520353] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3627.586917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3637.738592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.740462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3637.750763] Lustre: Skipped 1 previous similar message [ 3641.459548] LustreError: 133200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3641.463940] LustreError: 133200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3641.625149] Lustre: server umount lustre-OST0000 complete [ 3647.242551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.253812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.266786] Lustre: Skipped 1 previous similar message [ 3647.286395] Lustre: Skipped 2 previous similar messages [ 3652.578579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3652.591180] Lustre: Skipped 1 previous similar message [ 3653.545233] LustreError: 133449:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3653.555140] LustreError: 133449:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3654.078156] Lustre: server umount lustre-MDT0000 complete [ 3657.709516] LustreError: 128855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3657.742690] LustreError: 128855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3661.445712] LustreError: 128839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318922 with bad export cookie 3912354199260892537 [ 3661.470962] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.471361] LustreError: 128839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3661.779650] LustreError: 133700:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3661.807901] LustreError: 133700:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3662.707312] Lustre: server umount lustre-MDT0001 complete [ 3678.775471] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3681.700470] Key type lgssc unregistered [ 3682.032926] LNet: 134378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3682.041135] LNetError: 134378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3682.069848] LNet: Removed LNI 192.168.203.147@tcp [ 3683.132358] Key type .llcrypt unregistered [ 3683.134448] Key type ._llcrypt unregistered [ 3698.333830] Key type ._llcrypt registered [ 3698.338272] Key type .llcrypt registered [ 3698.550588] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3699.052350] Key type .llcrypt unregistered [ 3699.057251] Key type ._llcrypt unregistered [ 3784.788557] Key type ._llcrypt registered [ 3784.790448] Key type .llcrypt registered [ 3784.904741] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 3785.448707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3785.503145] alg: No test for adler32 (adler32-zlib) [ 3786.524672] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 3787.010513] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 3788.975173] Key type lgssc registered [ 3791.102237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.870769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3799.888381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.574889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.610455] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3811.825085] LustreError: 139301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.839468] LustreError: 139301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3816.938350] LustreError: 139301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3819.804221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3820.182225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.978883] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3831.070260] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3835.368590] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3846.158551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3846.549920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.242697] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 3858.819840] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3871.103959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3886.560052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.561974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.582731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.619022] Lustre: Skipped 1 previous similar message [ 3890.936823] LustreError: 141959:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3891.069339] Lustre: server umount lustre-OST0000 complete [ 3896.801402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3896.808079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.812421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.812429] Lustre: Skipped 1 previous similar message [ 3896.848603] Lustre: Skipped 1 previous similar message [ 3901.922396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.925993] Lustre: Skipped 1 previous similar message [ 3902.364339] LustreError: 142208:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3902.366394] LustreError: 142208:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3902.603426] Lustre: server umount lustre-MDT0000 complete [ 3906.017222] LustreError: 141237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.038369] LustreError: 141237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3907.567727] LustreError: 139281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319168 with bad export cookie 12947190370377106154 [ 3907.573094] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3907.584652] LustreError: 139281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3907.836982] LustreError: 142458:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3907.840475] LustreError: 142458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3908.340751] Lustre: server umount lustre-MDT0001 complete [ 3922.856740] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 3925.957656] Key type lgssc unregistered [ 3926.364112] LNet: 143135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3926.377564] LNetError: 143135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3926.392375] LNet: Removed LNI 192.168.203.147@tcp [ 3927.396208] Key type .llcrypt unregistered [ 3927.398456] Key type ._llcrypt unregistered [ 4012.668976] Key type ._llcrypt registered [ 4012.673857] Key type .llcrypt registered [ 4012.836983] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4014.144269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4014.179924] alg: No test for adler32 (adler32-zlib) [ 4015.313606] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 4015.523899] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 4017.159153] Key type lgssc registered [ 4018.281123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4026.565219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4026.588223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.176775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4032.962728] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4038.637485] LustreError: 147602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.664771] LustreError: 147602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4043.758326] LustreError: 147602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.063776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4045.361562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4050.320741] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4056.653943] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4060.541648] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4071.160755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4071.626816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4073.715322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 4078.377435] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4084.130678] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4094.432575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4094.438632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.468406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4095.458389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4095.459536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4095.478963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.507573] Lustre: Skipped 1 previous similar message [ 4099.945142] LustreError: 150235:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4100.092867] Lustre: server umount lustre-OST0000 complete [ 4105.699627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.714497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.719848] Lustre: Skipped 1 previous similar message [ 4110.817216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.822390] Lustre: Skipped 1 previous similar message [ 4115.939418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4115.944198] Lustre: Skipped 1 previous similar message [ 4119.007094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4119.151550] LustreError: 150485:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4119.157636] LustreError: 150485:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4119.277025] Lustre: server umount lustre-MDT0000 complete [ 4121.057075] LustreError: 148508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.073747] LustreError: 148508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4125.190863] LustreError: 147583:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319386 with bad export cookie 54686844179538113 [ 4125.203249] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4125.204634] LustreError: 147583:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4125.396596] LustreError: 150736:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4125.404920] LustreError: 150736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4125.629618] Lustre: server umount lustre-MDT0001 complete [ 4138.786820] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 4142.334530] Key type lgssc unregistered [ 4142.597253] LNet: 151414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4142.605694] LNetError: 151414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4142.617939] LNet: Removed LNI 192.168.203.147@tcp [ 4143.721407] Key type .llcrypt unregistered [ 4143.723943] Key type ._llcrypt unregistered [ 4229.944855] Key type ._llcrypt registered [ 4229.948714] Key type .llcrypt registered [ 4230.046849] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4231.500855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4231.537043] alg: No test for adler32 (adler32-zlib) [ 4232.748694] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 4232.974094] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 4234.672530] Key type lgssc registered [ 4236.153716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4245.999846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4246.020722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.562130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.433435] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4257.764535] LustreError: 155872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.790352] LustreError: 155872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4262.882436] LustreError: 155872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.699782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4266.155527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4271.129849] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4277.036491] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4281.838060] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4292.366634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4292.665693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.860389] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4307.896714] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4317.177367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 4330.989263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4330.991908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.017206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4331.491157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4331.503937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.517267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4332.525529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4332.533031] Lustre: Skipped 1 previous similar message [ 4335.452196] LustreError: 158534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4335.776572] Lustre: server umount lustre-OST0000 complete [ 4342.768650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.785139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.999885] LustreError: 158784:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4347.006064] LustreError: 158784:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4347.175839] Lustre: server umount lustre-MDT0000 complete [ 4347.872559] LustreError: 156778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.902775] LustreError: 156778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4351.934200] LustreError: 155852:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319613 with bad export cookie 7479013782771410500 [ 4351.939322] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4351.946978] LustreError: 155852:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4352.138054] LustreError: 159034:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4352.160212] LustreError: 159034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4352.341563] Lustre: server umount lustre-MDT0001 complete [ 4367.821168] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 4371.003212] Key type lgssc unregistered [ 4371.349975] LNet: 159711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4371.360943] LNetError: 159711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4371.378354] LNet: Removed LNI 192.168.203.147@tcp [ 4372.509326] Key type .llcrypt unregistered [ 4372.511225] Key type ._llcrypt unregistered [ 4452.877876] Key type ._llcrypt registered [ 4452.880312] Key type .llcrypt registered [ 4452.995216] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4453.585867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4453.672216] alg: No test for adler32 (adler32-zlib) [ 4454.711468] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 4454.891980] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 4456.543703] Key type lgssc registered [ 4457.882730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4466.498774] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4466.526504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4482.208924] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4487.327203] LustreError: 164138:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 4514.016641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.608854] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4527.290023] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4532.944320] LustreError: 164997:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4533.131568] Lustre: server umount lustre-OST0000 complete [ 4544.454563] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:10:04 (1767319804) [ 4558.075185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.823138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4563.936391] LustreError: 165862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.957747] LustreError: 165862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4564.433881] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4568.032952] LustreError: 165862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.176205] LustreError: 165862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.436673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4577.861259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.956853] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4588.965129] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4593.160432] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4603.869991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4604.289154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.910722] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4616.818748] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4628.866383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4648.416784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.437079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.463037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4649.441663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4649.445223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.105897] LustreError: 168823:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4652.115894] LustreError: 168823:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4652.317398] Lustre: server umount lustre-OST0000 complete [ 4659.687923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4659.693508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.718175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4659.718184] Lustre: Skipped 1 previous similar message [ 4659.784557] Lustre: Skipped 1 previous similar message [ 4663.434691] LustreError: 169072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4663.443298] LustreError: 169072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4663.579724] Lustre: server umount lustre-MDT0000 complete [ 4664.803246] LustreError: 165861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.839416] LustreError: 165861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4668.843574] LustreError: 165842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319929 with bad export cookie 16806165922272846726 [ 4668.873061] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.012672] LustreError: 169323:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4669.019522] LustreError: 169323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4669.261931] Lustre: server umount lustre-MDT0001 complete [ 4684.498494] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 4688.536622] Key type lgssc unregistered [ 4688.961735] LNet: 170000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4688.979399] LNetError: 170000:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4689.006806] LNet: Removed LNI 192.168.203.147@tcp [ 4690.217549] Key type .llcrypt unregistered [ 4690.222902] Key type ._llcrypt unregistered [ 4779.915682] Key type ._llcrypt registered [ 4779.918584] Key type .llcrypt registered [ 4780.087095] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 4781.351968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4781.424436] alg: No test for adler32 (adler32-zlib) [ 4782.673802] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 4783.034715] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 4784.719187] Key type lgssc registered [ 4786.027658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4794.642936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4794.677838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4796.213216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.776682] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4806.626949] LustreError: 174456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.658924] LustreError: 174456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4811.786149] LustreError: 174456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.558203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4813.975949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4818.442983] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4824.642279] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4828.886317] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4839.916887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4840.408842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.459773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4847.063908] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 4853.043549] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4867.042224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4867.046029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.053425] LustreError: Skipped 1 previous similar message [ 4867.075968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4869.090719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4869.092777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4872.585925] LustreError: 177109:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4873.061304] Lustre: server umount lustre-OST0000 complete [ 4879.328762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.329492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.346335] Lustre: Skipped 1 previous similar message [ 4879.357350] Lustre: Skipped 2 previous similar messages [ 4884.384112] LustreError: 177359:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4884.410915] LustreError: 177359:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4884.475149] LustreError: 174452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.489443] LustreError: 174452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4884.801088] Lustre: server umount lustre-MDT0000 complete [ 4889.512495] LustreError: 174436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320150 with bad export cookie 13879549271028823000 [ 4889.517107] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4889.543076] LustreError: 174436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4889.554515] LustreError: 173764:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000020abf8d9 x1853169382492416/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4889.732420] LustreError: 177609:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4889.736068] LustreError: 177609:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4890.050861] Lustre: server umount lustre-MDT0001 complete [ 4904.647901] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 4907.813871] Key type lgssc unregistered [ 4908.168744] LNet: 178286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4908.174717] LNetError: 178286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4909.228142] LNet: Removed LNI 192.168.203.147@tcp [ 4910.231778] Key type .llcrypt unregistered [ 4910.235290] Key type ._llcrypt unregistered [ 5001.398410] Key type ._llcrypt registered [ 5001.404610] Key type .llcrypt registered [ 5001.570174] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 5002.912032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5003.069666] alg: No test for adler32 (adler32-zlib) [ 5004.228194] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 5004.619335] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 5006.359628] Key type lgssc registered [ 5007.662188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5015.909579] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5015.931546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5031.647577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5036.767244] LustreError: 182955:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 5063.455683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.011254] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5076.266386] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5087.099357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.491525] LustreError: MGC192.168.203.147@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5087.506967] LustreError: 184060:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5087.513328] LustreError: 184060:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5087.528928] LustreError: 184060:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 5087.649411] Lustre: server umount lustre-MDT0000 complete [ 5087.651263] LustreError: 184060:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5108.703213] Lustre: 182294:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320353/real 1767320353] req@00000000d805afc4 x1853169614195968/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5108.732053] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5108.749644] LustreError: 182969:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 5129.188049] LustreError: 185135:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 5129.362536] Lustre: server umount lustre-OST0000 complete [ 5148.912617] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 5218.290903] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 5239.319746] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5253.536768] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5264.586284] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5275.463916] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5293.129714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5293.214427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.504325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5293.529898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5293.588131] Lustre: lustre-MDT0000: new disk, initializing [ 5293.659432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.680511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5298.825477] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5318.369755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5318.465920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5318.601677] Lustre: 192551:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5318.666966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5318.676198] Lustre: Skipped 1 previous similar message [ 5318.793510] Lustre: lustre-MDT0001: new disk, initializing [ 5318.963676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5319.018723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5319.043282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5324.565710] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5332.801266] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5336.990061] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5350.270993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5350.376511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5350.597556] Lustre: lustre-OST0000: new disk, initializing [ 5350.600259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5350.735036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.227699] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5358.168264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5358.195189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5358.309533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5365.378551] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5371.063908] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5371.350976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5376.543883] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5376.775048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5379.049740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.049959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.064794] LustreError: Skipped 1 previous similar message [ 5379.065481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5379.121227] Lustre: Skipped 1 previous similar message [ 5384.168848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5384.180386] Lustre: Skipped 2 previous similar messages [ 5389.282805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.290125] Lustre: Skipped 1 previous similar message [ 5393.375407] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5393.426810] LustreError: 196184:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5393.429730] LustreError: 196184:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5393.723734] Lustre: server umount lustre-OST0000 complete [ 5399.521209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.523773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.541236] Lustre: Skipped 1 previous similar message [ 5399.549468] Lustre: Skipped 1 previous similar message [ 5404.562443] LustreError: 196438:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5404.565988] LustreError: 196438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5404.702390] Lustre: server umount lustre-MDT0000 complete [ 5409.263977] LustreError: 192544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320670 with bad export cookie 11623251804274520282 [ 5409.267257] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5409.272599] LustreError: 192544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5409.437265] LustreError: 196687:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5409.450842] LustreError: 196687:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5409.595428] Lustre: server umount lustre-MDT0001 complete [ 5418.364766] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:24:39 (1767320679) [ 5422.485669] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 5458.020625] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 5521.859718] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 5543.096257] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5554.540788] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5564.401765] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5575.713319] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.903237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5591.946670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.073293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5592.086916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5592.132737] Lustre: lustre-MDT0000: new disk, initializing [ 5592.220381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.242286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5596.087490] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5612.977041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.070543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5613.233673] Lustre: 205589:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5613.320711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5613.324888] Lustre: Skipped 1 previous similar message [ 5613.453322] Lustre: lustre-MDT0001: new disk, initializing [ 5613.564157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5613.616524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5613.633132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5618.833622] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5626.404507] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5630.244992] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.929631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5644.031837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5644.277801] Lustre: lustre-OST0000: new disk, initializing [ 5644.294341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5644.422802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.685617] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5652.030812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5652.035484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5652.174786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5657.721029] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5662.403895] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5662.602309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5667.014445] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5667.228372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5672.935900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5672.944088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.947040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5672.952769] Lustre: Skipped 1 previous similar message [ 5672.960663] Lustre: Skipped 1 previous similar message [ 5683.167587] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5683.194443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5683.203659] Lustre: Skipped 3 previous similar messages [ 5683.287050] LustreError: 209222:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5683.290626] LustreError: 209222:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5683.504865] Lustre: server umount lustre-OST0000 complete [ 5688.296428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.322273] LustreError: 209473:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5694.328243] LustreError: 209473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5694.412626] Lustre: server umount lustre-MDT0000 complete [ 5698.529078] LustreError: 208878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5698.545376] LustreError: 208878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5699.273595] LustreError: 205581:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320960 with bad export cookie 11623251804274521164 [ 5699.279352] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.287903] LustreError: 205581:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5699.537794] Lustre: server umount lustre-MDT0001 complete [ 5711.074794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5711.456652] LustreError: 210391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.530946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.226443] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5716.967485] LustreError: 210391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.068174] LustreError: 210391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.182147] LustreError: 210391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.663242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5732.853625] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5737.860155] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5741.607598] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5752.170664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5759.694828] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 5765.896993] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5772.320453] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5777.376896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5777.392334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.399229] Lustre: Skipped 2 previous similar messages [ 5777.411454] Lustre: Skipped 5 previous similar messages [ 5782.855233] LustreError: 213120:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5782.864429] LustreError: 213120:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5783.218580] Lustre: server umount lustre-OST0000 complete [ 5789.664903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5789.671477] LustreError: Skipped 1 previous similar message [ 5789.675663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.220228] Lustre: server umount lustre-MDT0000 complete [ 5797.858262] LustreError: 210386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.872649] LustreError: 210386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5798.722260] LustreError: 210371:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321059 with bad export cookie 11623251804274522046 [ 5798.725216] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5798.732673] LustreError: 210371:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5799.129381] Lustre: server umount lustre-MDT0001 complete [ 5814.510586] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 5817.716554] Key type lgssc unregistered [ 5818.042745] LNet: 214297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5818.052819] LNetError: 214297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5818.071683] LNet: Removed LNI 192.168.203.147@tcp [ 5818.968746] Key type .llcrypt unregistered [ 5818.977046] Key type ._llcrypt unregistered [ 5857.442262] Key type ._llcrypt registered [ 5857.443665] Key type .llcrypt registered [ 5857.602374] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 5923.616701] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 5924.657763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5924.786059] alg: No test for adler32 (adler32-zlib) [ 5925.920033] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 5926.182163] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 5927.903161] Key type lgssc registered [ 5929.009746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5947.806969] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.212793] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5968.749939] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5977.605961] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5992.598254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5992.662211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5992.680319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.034643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5994.071824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5994.200830] Lustre: lustre-MDT0000: new disk, initializing [ 5994.356355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.394585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5998.866041] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6013.626590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6013.750501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6013.867403] Lustre: 223130:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6013.938094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6013.947980] Lustre: Skipped 1 previous similar message [ 6014.087132] Lustre: lustre-MDT0001: new disk, initializing [ 6014.278938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6014.332662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6014.344912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6019.238966] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6026.520945] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6030.591429] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6042.591087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6042.664092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6043.005463] Lustre: lustre-OST0000: new disk, initializing [ 6043.010760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6043.116815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6048.918226] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6049.895637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6049.911090] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6050.026217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6055.327918] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6059.374954] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6059.547158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6064.269692] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6064.520350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6070.754509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.757714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.757783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.764209] Lustre: Skipped 1 previous similar message [ 6072.448953] LustreError: 226761:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6072.716668] Lustre: server umount lustre-OST0000 complete [ 6080.998242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.002988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6081.004026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.004036] Lustre: Skipped 1 previous similar message [ 6081.008262] Lustre: Skipped 1 previous similar message [ 6083.668753] LustreError: 227010:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6083.674738] LustreError: 227010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6083.794873] Lustre: server umount lustre-MDT0000 complete [ 6088.504559] LustreError: 224255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321349 with bad export cookie 10933305460290081610 [ 6088.510384] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.512986] LustreError: 224255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6088.634909] LustreError: 227260:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6088.639256] LustreError: 227260:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6088.773980] Lustre: server umount lustre-MDT0001 complete [ 6096.305983] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:35:57 (1767321357) [ 6105.093881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.512957] LustreError: 228122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.533687] LustreError: 228122:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6105.607039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6109.559343] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6110.689448] LustreError: 228122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.812961] LustreError: 228122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.522450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6119.822768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6123.249842] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6128.249818] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6131.908133] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6135.267542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.272734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.278176] Lustre: Skipped 1 previous similar message [ 6135.287521] Lustre: Skipped 1 previous similar message [ 6135.290175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6140.385319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6140.390182] Lustre: Skipped 2 previous similar messages [ 6144.486357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6144.499488] Lustre: Skipped 1 previous similar message [ 6149.090074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6149.174942] LustreError: 229708:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6149.181539] LustreError: 229708:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6149.228125] Lustre: server umount lustre-MDT0000 complete [ 6149.601682] LustreError: 229484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6149.623950] LustreError: 229484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6153.481594] LustreError: 228104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321414 with bad export cookie 10933305460290082492 [ 6153.488580] LustreError: 228104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6153.491979] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6153.690324] LustreError: 229959:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6153.696382] LustreError: 229959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6154.051734] Lustre: server umount lustre-MDT0001 complete [ 6162.643380] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:37:03 (1767321423) [ 6173.751094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6189.348319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6194.463470] LustreError: 230795:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 6220.196256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.483051] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6233.147459] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6235.221119] LustreError: 231461:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6235.226336] LustreError: 231461:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6235.516437] Lustre: server umount lustre-OST0000 complete [ 6243.797443] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:38:24 (1767321504) [ 6254.576474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.017022] LustreError: 232325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6255.087358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6259.300797] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6265.315565] LustreError: 232324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.330085] LustreError: 232324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6270.717528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6275.436870] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6280.669641] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6284.273410] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6294.862331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6295.289929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.301519] Lustre: Skipped 1 previous similar message [ 6301.146981] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6305.788031] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6316.003522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.011664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6316.020844] Lustre: Skipped 2 previous similar messages [ 6316.046171] Lustre: Skipped 2 previous similar messages [ 6319.719794] LustreError: 235003:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6319.727742] LustreError: 235003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6319.891592] Lustre: server umount lustre-MDT0000 complete [ 6321.122265] LustreError: 233230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.136718] LustreError: 233230:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6323.277983] LustreError: 235004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321584 with bad export cookie 10933305460290083150 [ 6323.279478] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6323.292315] LustreError: 235004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6323.559975] Lustre: server umount lustre-MDT0001 complete [ 6337.076973] Lustre: server umount lustre-OST0000 complete [ 6345.931756] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:40:06 (1767321606) [ 6356.553888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.034203] LustreError: 236319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.705922] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6374.300080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6374.630798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6374.634975] Lustre: Skipped 1 previous similar message [ 6379.394705] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6385.392048] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6389.972889] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6401.204040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6402.541821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 6407.829454] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6413.301466] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6418.097967] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6418.314599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6422.306834] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6422.593251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6425.056818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.070467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.072647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.082433] Lustre: Skipped 1 previous similar message [ 6426.591181] LustreError: 239451:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6426.592852] LustreError: 239451:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6426.682209] Lustre: server umount lustre-OST0000 complete [ 6435.301098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6438.223426] Lustre: server umount lustre-MDT0000 complete [ 6440.416874] LustreError: 236318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.437533] LustreError: 236318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6442.295695] LustreError: 237233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321703 with bad export cookie 10933305460290084228 [ 6442.304574] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.306551] LustreError: 237233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6442.585448] Lustre: server umount lustre-MDT0001 complete [ 6449.609892] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:41:51 (1767321711) [ 6458.519940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6474.079935] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6479.200951] LustreError: 240786:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 6504.929892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.938109] Lustre: Skipped 1 previous similar message [ 6511.008315] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6516.904826] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6526.863160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6527.355507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 6531.338865] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6542.213308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6547.161887] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6552.296814] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6555.847239] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6559.554705] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6559.775119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6564.032229] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6564.229622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6568.422245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.454101] Lustre: Skipped 1 previous similar message [ 6568.459530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.469232] Lustre: Skipped 4 previous similar messages [ 6572.713280] LustreError: 243998:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6572.720272] LustreError: 243998:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6572.798111] Lustre: server umount lustre-MDT0000 complete [ 6573.538661] LustreError: 240813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.555187] LustreError: 240813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6576.916090] LustreError: 240793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321837 with bad export cookie 10933305460290085250 [ 6576.926097] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6576.930556] LustreError: 240793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6577.305802] Lustre: server umount lustre-MDT0001 complete [ 6591.068925] Lustre: server umount lustre-OST0000 complete [ 6598.099474] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:44:19 (1767321859) [ 6605.979842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6621.536729] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6626.655447] LustreError: 245285:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 6658.703255] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6664.070423] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6673.984360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6674.690075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6678.925389] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6690.553393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6694.914651] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6701.527308] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6705.023889] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6717.584079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6717.613475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6717.668519] Lustre: lustre-OST0001: new disk, initializing [ 6717.670583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6723.737708] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6725.730490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6725.755506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6725.940936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6733.050977] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6738.972319] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6739.297917] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6744.063694] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6744.197225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6746.599790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.624473] Lustre: Skipped 2 previous similar messages [ 6746.632259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6746.635571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.398828] LustreError: 249754:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6752.401755] LustreError: 249754:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6752.637900] Lustre: server umount lustre-OST0000 complete [ 6756.836701] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6756.852777] LustreError: Skipped 1 previous similar message [ 6763.124762] Lustre: server umount lustre-OST0001 complete [ 6774.104929] Lustre: server umount lustre-MDT0000 complete [ 6778.511269] LustreError: 245293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322039 with bad export cookie 10933305460290086195 [ 6778.516569] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.527867] LustreError: 245293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6779.006341] Lustre: server umount lustre-MDT0001 complete [ 6784.968208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6789.394300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6793.674340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6797.993330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6804.614740] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:47:45 (1767322065) [ 6809.107300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6810.659406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6812.140494] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:47:53 (1767322073) [ 6822.394512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.933702] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6822.971746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6823.234089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6823.239658] Lustre: Skipped 6 previous similar messages [ 6827.550685] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6839.401420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6839.483704] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6839.516644] Lustre: 252621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6844.190930] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6849.234393] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6853.084080] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6863.094091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6863.334388] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6869.186993] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6871.196124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6875.121200] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6879.504549] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6879.674911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6884.171583] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6884.468404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6886.373062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6886.381112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.400476] Lustre: Skipped 5 previous similar messages [ 6886.416919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6886.422910] Lustre: Skipped 9 previous similar messages [ 6892.414245] Lustre: server umount lustre-OST0000 complete [ 6917.021775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6918.314481] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6918.329582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6918.342808] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6918.370075] Lustre: Skipped 1 previous similar message [ 6922.827566] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 6927.870611] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6932.500031] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6932.752400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6936.190280] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6936.456204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6939.603324] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96990579d800.ost_server_uuid 50 [ 6940.841572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96990579d800.ost_server_uuid in FULL state after 0 sec [ 6941.045160] LustreError: 252631:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.47@tcp) returned error from blocking AST (req@00000000259a3448 x1853170581233408 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ccd697ad/0x97bae763956dee86 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.47@tcp remote: 0xcdc631a26929a26c expref: 5 pid: 252631 timeout: 7041 lvb_type: 0 [ 6941.094793] LustreError: lustre-MDT0000: A client on nid 192.168.203.47@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6941.113171] LustreError: 252619:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.47@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ccd697ad/0x97bae763956dee86 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.47@tcp remote: 0xcdc631a26929a26c expref: 6 pid: 252631 timeout: 0 lvb_type: 0 [ 6946.789820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6946.799021] LustreError: Skipped 1 previous similar message [ 6951.111937] Lustre: server umount lustre-OST0000 complete [ 6960.959586] Lustre: server umount lustre-MDT0000 complete [ 6962.659965] LustreError: 252630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6962.683880] LustreError: 252630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6966.319408] LustreError: 252615:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322227 with bad export cookie 10933305460290087371 [ 6966.321907] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6966.328286] LustreError: 252615:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6972.762972] Lustre: server umount lustre-MDT0001 complete [ 6986.557994] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 6988.902280] Key type lgssc unregistered [ 6989.415942] LNet: 258933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6989.424275] LNetError: 258933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6989.444049] LNet: Removed LNI 192.168.203.147@tcp [ 6990.427886] Key type .llcrypt unregistered [ 6990.431351] Key type ._llcrypt unregistered [ 7060.591672] Key type ._llcrypt registered [ 7060.594461] Key type .llcrypt registered [ 7060.691382] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7061.485432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7061.649877] alg: No test for adler32 (adler32-zlib) [ 7062.702157] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7062.920416] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7064.559185] Key type lgssc registered [ 7065.459935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7072.723837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7072.747678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.311349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.401626] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7084.523065] LustreError: 263387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.546833] LustreError: 263387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7088.996536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7089.334867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7093.513565] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7098.376873] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7101.910556] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7112.346496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7112.627554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.751778] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7119.802979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7122.797303] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7128.981111] Lustre: Failing over lustre-MDT0000 [ 7129.143459] LustreError: 266018:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7129.238802] Lustre: server umount lustre-MDT0000 complete [ 7130.082831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.087756] LustreError: 264292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7130.090682] Lustre: Skipped 1 previous similar message [ 7130.105574] LustreError: 264292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7135.203413] LustreError: 263382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.212110] LustreError: 263382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7140.323891] LustreError: 263387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.090864] LustreError: 265392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.47@tcp arrived at 1767322402 with bad export cookie 4842443122814140850 [ 7145.441664] LustreError: 263387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.458560] LustreError: 263387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7146.463172] Lustre: 262697:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322391/real 1767322391] req@00000000d05d8334 x1853171772185728/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767322407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7146.487172] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7150.080666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.706159] LustreError: 262695:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000108d8d57 x1853171772190848/t0(0) o250->MGC192.168.203.147@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7156.737040] LustreError: 263383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7156.757363] LustreError: 263383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7157.068054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7157.135550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7160.999667] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7162.342284] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7162.349144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7162.948329] Lustre: lustre-MDT0000: Denying connection for new client 171de551-cb0f-4238-9d53-98d5074a582a (at 192.168.203.47@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7171.040375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7171.047920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7171.060832] Lustre: Skipped 1 previous similar message [ 7171.073766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.576754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.578325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.643161] LustreError: 267146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7176.646233] LustreError: 267146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7176.795802] Lustre: server umount lustre-OST0000 complete [ 7181.017873] LustreError: 267395:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 7181.024490] Lustre: 266711:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 7181.030646] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 7181.044857] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 7181.052485] LustreError: 266711:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004682f4c5 x1853171772205824/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7181.067779] LustreError: 266711:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7181.076710] LustreError: 266711:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 7181.086416] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 7181.270856] LustreError: 267395:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7181.277127] LustreError: 267395:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7181.408080] Lustre: server umount lustre-MDT0000 complete [ 7182.818948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7182.835644] LustreError: 266134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.591207] LustreError: 263365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322446 with bad export cookie 4842443122814140948 [ 7185.603047] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.604771] LustreError: 263365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7187.764654] LustreError: 267646:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7187.771354] LustreError: 267646:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7187.952909] Lustre: server umount lustre-MDT0001 complete [ 7200.372652] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7202.819489] Key type lgssc unregistered [ 7203.098153] LNet: 268323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7203.104066] LNetError: 268323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7203.118182] LNet: Removed LNI 192.168.203.147@tcp [ 7203.752567] Key type .llcrypt unregistered [ 7203.756344] Key type ._llcrypt unregistered [ 7277.682748] Key type ._llcrypt registered [ 7277.685337] Key type .llcrypt registered [ 7277.768770] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7278.574748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7278.616221] alg: No test for adler32 (adler32-zlib) [ 7279.678207] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7279.882654] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7281.567295] Key type lgssc registered [ 7282.464794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7288.818527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7288.836909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.183719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.584925] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7300.586259] LustreError: 272777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7300.616582] LustreError: 272777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7303.278045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7303.610816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7307.509384] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7312.250365] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7315.448073] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7325.195301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7325.524656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.383849] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7335.359663] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7336.824801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7347.170169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7347.174060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.189823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7351.226621] LustreError: 275412:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7351.377592] Lustre: server umount lustre-OST0000 complete [ 7357.410777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.426038] Lustre: Skipped 1 previous similar message [ 7357.434210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7357.436362] Lustre: Skipped 1 previous similar message [ 7361.448971] LustreError: 275663:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7361.456869] LustreError: 275663:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7361.570780] Lustre: server umount lustre-MDT0000 complete [ 7362.529697] LustreError: 272778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.553257] LustreError: 272778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7365.892907] LustreError: 272758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322626 with bad export cookie 9362913483798218078 [ 7365.893976] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.899067] LustreError: 272758:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7366.048184] LustreError: 275913:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7366.060928] LustreError: 275913:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7366.429820] Lustre: server umount lustre-MDT0001 complete [ 7380.296833] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7382.712634] Key type lgssc unregistered [ 7382.942845] LNet: 276590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7382.952522] LNetError: 276590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7382.977311] LNet: Removed LNI 192.168.203.147@tcp [ 7383.806887] Key type .llcrypt unregistered [ 7383.808548] Key type ._llcrypt unregistered [ 7464.151965] Key type ._llcrypt registered [ 7464.153746] Key type .llcrypt registered [ 7464.218929] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7465.067953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7465.101416] alg: No test for adler32 (adler32-zlib) [ 7466.074823] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7466.248519] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7467.911184] Key type lgssc registered [ 7468.789342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7475.213501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7475.221957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.558991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.117581] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7486.952929] LustreError: 281719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.979647] LustreError: 281719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7489.352725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7489.623139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7492.712727] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7496.933674] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7499.801946] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7507.551126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7507.907338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.833713] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7517.326355] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7522.302588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7527.392773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.396156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7527.407357] Lustre: Skipped 1 previous similar message [ 7527.408405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7528.018127] LustreError: 284348:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7528.110156] Lustre: server umount lustre-OST0000 complete [ 7532.525152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.533188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7532.545098] Lustre: Skipped 1 previous similar message [ 7532.567465] Lustre: Skipped 2 previous similar messages [ 7537.638717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7537.649684] Lustre: Skipped 1 previous similar message [ 7542.752150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.755501] Lustre: Skipped 1 previous similar message [ 7546.335155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7546.401754] LustreError: 284597:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7546.406840] LustreError: 284597:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7546.449159] Lustre: server umount lustre-MDT0000 complete [ 7547.872375] LustreError: 282626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.881787] LustreError: 282626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7549.450902] LustreError: 281698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322810 with bad export cookie 10477917078582804471 [ 7549.461048] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.464656] LustreError: 281698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7549.565096] LustreError: 284850:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7549.568577] LustreError: 284850:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7549.679440] Lustre: server umount lustre-MDT0001 complete [ 7557.485194] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7559.120425] Key type lgssc unregistered [ 7559.337792] LNet: 285525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7559.344907] LNetError: 285525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7559.356408] LNet: Removed LNI 192.168.203.147@tcp [ 7559.896078] Key type .llcrypt unregistered [ 7559.898249] Key type ._llcrypt unregistered [ 7617.285615] Key type ._llcrypt registered [ 7617.287338] Key type .llcrypt registered [ 7617.337762] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7617.944395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7618.000307] alg: No test for adler32 (adler32-zlib) [ 7618.965513] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7619.135113] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7620.768090] Key type lgssc registered [ 7621.475958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7629.806237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7629.819541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.066360] Lustre: *** cfs_fail_loc=135, val=0*** [ 7631.119329] LustreError: 290098:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 7631.126146] LustreError: 290098:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.203.147@tcp: cfg command failed: rc = -2 [ 7631.134049] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7631.134049] [ 7631.142495] LustreError: MGC192.168.203.147@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7631.151714] LustreError: 290058:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7631.157618] LustreError: 290058:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 7631.162827] LustreError: 290058:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 7631.167172] LustreError: 290058:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 7631.172982] LustreError: 290058:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7631.239439] Lustre: server umount lustre-MDT0000 complete [ 7631.247301] LustreError: 290058:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7637.210267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.532372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.128890] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7647.714273] LustreError: 290682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.726487] LustreError: 290682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7648.623446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7648.865736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.593635] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7655.316708] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7657.881853] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7663.077169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.080856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7663.088793] Lustre: Skipped 1 previous similar message [ 7663.100434] Lustre: Skipped 1 previous similar message [ 7665.747803] LustreError: 292247:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7665.754650] LustreError: 292247:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7665.885606] Lustre: server umount lustre-MDT0000 complete [ 7668.198264] LustreError: 290682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.221145] LustreError: 290682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7669.795047] LustreError: 290645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322930 with bad export cookie 1959174311724962774 [ 7669.797148] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7669.809556] LustreError: 290645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7669.914092] LustreError: 292497:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7669.916780] LustreError: 292497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7670.161654] Lustre: server umount lustre-MDT0001 complete [ 7679.826063] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7681.833669] Key type lgssc unregistered [ 7682.073293] LNet: 293175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7682.079699] LNetError: 293175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7682.090898] LNet: Removed LNI 192.168.203.147@tcp [ 7682.716957] Key type .llcrypt unregistered [ 7682.719815] Key type ._llcrypt unregistered [ 7700.672819] Key type ._llcrypt registered [ 7700.674802] Key type .llcrypt registered [ 7700.737371] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7700.984241] Key type .llcrypt unregistered [ 7700.986143] Key type ._llcrypt unregistered [ 7747.042725] Key type ._llcrypt registered [ 7747.045346] Key type .llcrypt registered [ 7747.121526] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7747.905164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7747.914756] alg: No test for adler32 (adler32-zlib) [ 7748.875113] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7749.019737] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7750.639156] Key type lgssc registered [ 7751.503828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7756.732097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7756.743684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7772.255655] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7777.376319] LustreError: 298565:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.147@tcp: failed processing log, type 4: rc = -110 [ 7803.999176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.610429] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7811.027952] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7818.211640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.676102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.731862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7821.907324] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7823.847776] LustreError: 298590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7823.869333] LustreError: 298590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7828.962663] LustreError: 298590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7830.497196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7830.807029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7833.767851] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7837.428171] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7840.287391] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7842.171634] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7854.569260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7854.576320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.586576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7856.610381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7856.616857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.631656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.664830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.673234] Lustre: Skipped 1 previous similar message [ 7860.339594] LustreError: 301721:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7860.447848] Lustre: server umount lustre-OST0000 complete [ 7866.848242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7866.854959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7866.864784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.899455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.058103] LustreError: 301971:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7870.066498] LustreError: 301971:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7870.273422] Lustre: server umount lustre-MDT0000 complete [ 7874.016101] LustreError: 299680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.041961] LustreError: 299680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7874.464479] LustreError: 298573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323135 with bad export cookie 14156697910831104113 [ 7874.480030] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7874.484720] LustreError: 298573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7874.627361] LustreError: 302220:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7874.638028] LustreError: 302220:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7874.806837] Lustre: server umount lustre-MDT0001 complete [ 7886.458563] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 7888.292640] Key type lgssc unregistered [ 7888.473387] LNet: 302897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7888.475852] LNetError: 302897:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7888.489897] LNet: Removed LNI 192.168.203.147@tcp [ 7888.912182] Key type .llcrypt unregistered [ 7888.913494] Key type ._llcrypt unregistered [ 7952.774453] Key type ._llcrypt registered [ 7952.780032] Key type .llcrypt registered [ 7952.905349] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 7953.736098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7953.756791] alg: No test for adler32 (adler32-zlib) [ 7954.773574] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 7955.006620] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 7956.687160] Key type lgssc registered [ 7957.558490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7963.188488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7963.205805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.554129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7967.178830] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7969.766853] LustreError: 307492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7969.794365] LustreError: 307492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7974.881889] LustreError: 307492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7976.027893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7976.347359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7979.517144] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7983.178887] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7985.635323] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7991.869771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7992.095903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7996.031648] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 7999.734425] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8009.513854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 8013.802349] Lustre: Failing over lustre-MDT0000 [ 8013.912423] LustreError: 310196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8013.973573] Lustre: server umount lustre-MDT0000 complete [ 8014.820340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.832648] LustreError: 307489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.837337] Lustre: Skipped 2 previous similar messages [ 8014.846613] LustreError: 307489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8019.940112] LustreError: 307493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.964538] LustreError: 307493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8025.057879] LustreError: 308764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8025.072039] LustreError: 308764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8030.175170] Lustre: 306802:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767323275/real 1767323275] req@00000000515e672c x1853172708564224/t0(0) o400->MGC192.168.203.147@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767323291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8030.215305] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8035.296954] LustreError: 307493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.312264] LustreError: 307493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 8037.807795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8040.422364] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5bd719440d825269 [ 8040.431337] Lustre: MGC192.168.203.147@tcp: Connection restored to 0@lo (at 0@lo) [ 8040.679458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.738444] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8043.093339] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8043.555721] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8046.059653] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8046.077448] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8046.099506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 8046.477089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 8047.700848] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 8056.288801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8056.295137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.309659] Lustre: Skipped 1 previous similar message [ 8056.310332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8056.318082] Lustre: Skipped 1 previous similar message [ 8060.468095] LustreError: 311917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8060.471913] LustreError: 311917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8060.628150] Lustre: server umount lustre-OST0000 complete [ 8066.528579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8066.534034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8066.539860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8066.543620] Lustre: Skipped 1 previous similar message [ 8066.559447] Lustre: Skipped 1 previous similar message [ 8070.390508] LustreError: 312167:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8070.403942] LustreError: 312167:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8070.538919] Lustre: server umount lustre-MDT0000 complete [ 8071.648834] LustreError: 307488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.673631] LustreError: 307488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8074.062230] LustreError: 307473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323335 with bad export cookie 6617785957519348329 [ 8074.065057] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.082191] LustreError: 307473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8074.194797] LustreError: 312416:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8074.202569] LustreError: 312416:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8074.323825] Lustre: server umount lustre-MDT0001 complete [ 8083.824820] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8085.825392] Key type lgssc unregistered [ 8086.073722] LNet: 313092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.080359] LNetError: 313092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.105038] LNet: Removed LNI 192.168.203.147@tcp [ 8086.741431] Key type .llcrypt unregistered [ 8086.743340] Key type ._llcrypt unregistered [ 8143.410451] Key type ._llcrypt registered [ 8143.413535] Key type .llcrypt registered [ 8143.486711] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8144.146332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8144.215439] alg: No test for adler32 (adler32-zlib) [ 8145.168826] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8145.310996] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8146.919140] Key type lgssc registered [ 8147.594127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8152.660340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8152.682628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.007521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.652691] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8164.320544] LustreError: 317688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8164.327616] LustreError: 317688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8164.825110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8165.068826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8168.123571] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8172.050304] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8174.367798] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8181.213137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8181.467362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.119624] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8188.478643] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8199.914451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 8201.492485] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8207.609698] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 8217.157712] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8225.766690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8225.770983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.776156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8225.794747] Lustre: Skipped 1 previous similar message [ 8230.336418] LustreError: 320501:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8230.466932] Lustre: server umount lustre-OST0000 complete [ 8236.004224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.015737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8236.018613] Lustre: Skipped 2 previous similar messages [ 8239.564202] LustreError: 320755:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8239.568781] LustreError: 320755:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8239.658431] Lustre: server umount lustre-MDT0000 complete [ 8241.120500] LustreError: 317682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.132040] LustreError: 317682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8242.702832] LustreError: 317668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323503 with bad export cookie 9075920591107605238 [ 8242.709796] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8242.823965] LustreError: 321005:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8242.827442] LustreError: 321005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8242.975902] Lustre: server umount lustre-MDT0001 complete [ 8251.169993] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8252.910922] Key type lgssc unregistered [ 8253.101768] LNet: 321680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8253.107613] LNetError: 321680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8253.122841] LNet: Removed LNI 192.168.203.147@tcp [ 8253.599655] Key type .llcrypt unregistered [ 8253.602123] Key type ._llcrypt unregistered [ 8300.594331] Key type ._llcrypt registered [ 8300.597451] Key type .llcrypt registered [ 8300.651529] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8301.175106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8301.186182] alg: No test for adler32 (adler32-zlib) [ 8302.158791] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8302.286959] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8303.895162] Key type lgssc registered [ 8304.601166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8309.095205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8309.110737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.382649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.776161] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8319.751201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8319.970354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8322.145168] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8325.144321] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8326.991686] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8332.292230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8332.522260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.466543] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8336.623430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8338.328876] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8342.954453] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8354.482608] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8364.613766] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8372.664705] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8380.897919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.898126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8380.910269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8380.914760] Lustre: Skipped 1 previous similar message [ 8386.016934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8386.020848] Lustre: Skipped 1 previous similar message [ 8386.606201] LustreError: 330731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8386.654689] Lustre: server umount lustre-OST0000 complete [ 8391.138640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8391.142677] Lustre: Skipped 1 previous similar message [ 8391.145167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.151029] Lustre: Skipped 1 previous similar message [ 8395.346375] LustreError: 330980:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8395.349173] LustreError: 330980:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8395.393552] Lustre: server umount lustre-MDT0000 complete [ 8396.260481] LustreError: 326269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8396.274816] LustreError: 326269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8398.264866] LustreError: 326254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323659 with bad export cookie 1491196191209929058 [ 8398.269257] LustreError: 326254:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8398.270502] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8398.359804] LustreError: 331229:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8398.362465] LustreError: 331229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8398.450553] Lustre: server umount lustre-MDT0001 complete [ 8406.293274] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8408.070509] Key type lgssc unregistered [ 8408.266592] LNet: 331906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8408.271658] LNetError: 331906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8408.293978] LNet: Removed LNI 192.168.203.147@tcp [ 8408.772848] Key type .llcrypt unregistered [ 8408.774079] Key type ._llcrypt unregistered [ 8451.647588] Key type ._llcrypt registered [ 8451.649563] Key type .llcrypt registered [ 8451.731811] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8452.247334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8452.336418] alg: No test for adler32 (adler32-zlib) [ 8453.268492] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8453.374792] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8454.999181] Key type lgssc registered [ 8455.599087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8459.281803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8459.289856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8460.521397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.489233] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8468.627181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8468.791428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8470.455690] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8472.869123] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8474.654232] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8479.251421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8479.422262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.787248] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8484.166800] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8485.609901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8496.095874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8496.098578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.103954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.193591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8500.196857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.201784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.206641] Lustre: Skipped 1 previous similar message [ 8501.485393] LustreError: 339280:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8501.547868] Lustre: server umount lustre-OST0000 complete [ 8504.804502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8504.808651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.814741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8509.766491] LustreError: 339530:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8509.769992] LustreError: 339530:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8509.822605] Lustre: server umount lustre-MDT0000 complete [ 8510.434622] LustreError: 336501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8510.441022] LustreError: 336501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8512.016729] LustreError: 339533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323773 with bad export cookie 4012138217447687368 [ 8512.022666] LustreError: 339533:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8512.022882] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8512.104535] LustreError: 339782:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8512.107213] LustreError: 339782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8512.199357] Lustre: server umount lustre-MDT0001 complete [ 8517.927370] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8519.254351] Key type lgssc unregistered [ 8519.426832] LNet: 340457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8519.430099] LNetError: 340457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8519.441448] LNet: Removed LNI 192.168.203.147@tcp [ 8519.794246] Key type .llcrypt unregistered [ 8519.796193] Key type ._llcrypt unregistered [ 8555.665762] Key type ._llcrypt registered [ 8555.667433] Key type .llcrypt registered [ 8555.713167] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8556.186935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8556.197896] alg: No test for adler32 (adler32-zlib) [ 8557.101341] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8557.203904] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8558.807383] Key type lgssc registered [ 8559.293615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8562.453857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8562.459753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.688613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.247642] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8570.530486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8570.707425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.415686] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8574.806435] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8576.444236] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8580.778373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8580.940559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8583.677708] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8586.213653] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8588.134313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8598.495795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8598.498447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.504051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.546494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8600.546775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8600.549903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.557369] Lustre: Skipped 1 previous similar message [ 8602.350110] LustreError: 347685:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8602.393878] Lustre: server umount lustre-OST0000 complete [ 8605.665375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8605.667141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8605.670844] Lustre: Skipped 1 previous similar message [ 8610.688332] LustreError: 347935:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8610.690667] LustreError: 347935:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8610.719100] Lustre: server umount lustre-MDT0000 complete [ 8610.784229] LustreError: 344904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.793471] LustreError: 344904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8612.949864] LustreError: 344884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323874 with bad export cookie 471050160038354389 [ 8612.955995] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8612.958121] LustreError: 344884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8613.056275] LustreError: 348184:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8613.065101] LustreError: 348184:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8613.141482] Lustre: server umount lustre-MDT0001 complete [ 8619.436851] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8620.802634] Key type lgssc unregistered [ 8620.977895] LNet: 348860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8620.982650] LNetError: 348860:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8620.993584] LNet: Removed LNI 192.168.203.147@tcp [ 8621.457913] Key type .llcrypt unregistered [ 8621.459524] Key type ._llcrypt unregistered [ 8657.184883] Key type ._llcrypt registered [ 8657.186570] Key type .llcrypt registered [ 8657.240661] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8657.639659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8657.693466] alg: No test for adler32 (adler32-zlib) [ 8658.582225] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8658.679581] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8660.271156] Key type lgssc registered [ 8660.716367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8664.211236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8664.220571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.450430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.278743] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8673.031218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8673.242056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8674.917769] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8677.080539] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8678.744867] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8683.290686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8683.427058] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8683.472180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8686.177456] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8688.490672] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8690.663545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8698.286390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8698.323558] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8698.426279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8700.888630] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8703.281940] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8706.103548] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 8716.905149] Lustre: Permanently deactivating lustre-OST0001 [ 8716.913043] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8724.483772] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8730.750303] Lustre: Permanently reactivating lustre-OST0001 [ 8730.760990] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8730.764985] Lustre: Skipped 2 previous similar messages [ 8731.718947] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8731.724346] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8731.728079] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8731.733368] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8731.752873] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8741.348878] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.353376] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8741.356093] Lustre: Skipped 2 previous similar messages [ 8743.665792] LustreError: 357379:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8743.688638] Lustre: server umount lustre-OST0001 complete [ 8746.465698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.466405] LustreError: 355847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8746.467929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8746.467935] Lustre: Skipped 1 previous similar message [ 8746.475768] Lustre: Skipped 1 previous similar message [ 8746.483353] LustreError: 355847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8751.468197] LustreError: 357580:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8751.470386] LustreError: 357580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8751.510087] Lustre: server umount lustre-OST0000 complete [ 8755.168856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8755.173178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8755.178840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8755.181872] Lustre: Skipped 1 previous similar message [ 8759.562057] LustreError: 357831:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8759.564227] LustreError: 357831:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8759.603430] Lustre: server umount lustre-MDT0000 complete [ 8761.593976] LustreError: 353286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324022 with bad export cookie 7063680904087024120 [ 8761.597498] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8761.598809] LustreError: 353286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8761.684708] LustreError: 358080:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8761.687596] LustreError: 358080:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8761.748397] Lustre: server umount lustre-MDT0001 complete [ 8767.316400] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 8768.705352] Key type lgssc unregistered [ 8768.851448] LNet: 358756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8768.856313] LNetError: 358756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8768.868691] LNet: Removed LNI 192.168.203.147@tcp [ 8769.230520] Key type .llcrypt unregistered [ 8769.231833] Key type ._llcrypt unregistered [ 8794.466185] Key type ._llcrypt registered [ 8794.467400] Key type .llcrypt registered [ 8794.506669] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 8825.398355] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 8825.891098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8825.924758] alg: No test for adler32 (adler32-zlib) [ 8826.797717] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 8826.903532] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 8828.503294] Key type lgssc registered [ 8829.083131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8837.572929] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8842.509560] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8846.166819] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.275764] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8856.477949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8856.501229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8856.506654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.624478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8857.643801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8857.684680] Lustre: lustre-MDT0000: new disk, initializing [ 8857.723248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8857.731064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8859.474618] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8867.595034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8867.624385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8867.663624] Lustre: 368774:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8867.688388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8867.690682] Lustre: Skipped 1 previous similar message [ 8867.728255] Lustre: lustre-MDT0001: new disk, initializing [ 8867.769696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8867.789684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8867.795626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8869.413235] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8872.997895] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8874.605643] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8880.159490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8880.188910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8880.301777] Lustre: lustre-OST0000: new disk, initializing [ 8880.304641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8880.344131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8881.022542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8881.026030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8881.067782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8882.519222] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8886.130689] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8887.818983] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8887.900324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8889.477480] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8889.555393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8891.363842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.364030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.370640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.375199] Lustre: Skipped 1 previous similar message [ 8896.431126] LustreError: 372401:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8896.490302] Lustre: server umount lustre-OST0000 complete [ 8898.528150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8898.533434] LustreError: Skipped 1 previous similar message [ 8898.536625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8898.542334] Lustre: Skipped 1 previous similar message [ 8898.544729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8901.601125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8901.601750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8901.609138] Lustre: Skipped 1 previous similar message [ 8904.591733] LustreError: 372651:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8904.596624] LustreError: 372651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8904.635514] Lustre: server umount lustre-MDT0000 complete [ 8906.628208] LustreError: 369897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324167 with bad export cookie 11199557766018949524 [ 8906.632741] LustreError: 369897:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8906.640862] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8906.725773] LustreError: 372900:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8906.728681] LustreError: 372900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8906.834934] Lustre: server umount lustre-MDT0001 complete [ 8910.732780] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 22:22:52 (1767324172) [ 8917.731268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.018723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8919.985215] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8923.105731] LustreError: 373906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.109888] LustreError: 373906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8926.294745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8926.482671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8928.178649] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8930.435140] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8931.993714] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8936.625085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8936.675455] LustreError: 375889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.791307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.227502] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 8941.550396] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8948.439256] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 8952.052339] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 8959.776444] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 8969.575960] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8981.740561] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8981.744359] Lustre: Skipped 1 previous similar message [ 9003.998817] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9004.004509] Lustre: Skipped 2 previous similar messages [ 9044.716994] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9044.722571] Lustre: Skipped 4 previous similar messages [ 9114.053340] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 9114.063081] Lustre: Skipped 7 previous similar messages [ 9117.154217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.157827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9117.164147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9117.171497] Lustre: Skipped 1 previous similar message [ 9122.273393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9122.275631] Lustre: Skipped 1 previous similar message [ 9122.289116] LustreError: 377805:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9122.292218] LustreError: 377805:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9122.338386] Lustre: server umount lustre-OST0000 complete [ 9126.369575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9126.372285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.377039] Lustre: Skipped 1 previous similar message [ 9130.437274] LustreError: 378055:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9130.439714] LustreError: 378055:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9130.473097] Lustre: server umount lustre-MDT0000 complete [ 9132.520458] LustreError: 373907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.540157] LustreError: 373907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9132.665195] LustreError: 376707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324393 with bad export cookie 11199557766018950406 [ 9132.667875] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9132.669979] LustreError: 376707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9132.846991] Lustre: server umount lustre-MDT0001 complete [ 9139.330986] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9140.653206] Key type lgssc unregistered [ 9140.808679] LNet: 378979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9140.812637] LNetError: 378979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9140.822693] LNet: Removed LNI 192.168.203.147@tcp [ 9141.246190] Key type .llcrypt unregistered [ 9141.247656] Key type ._llcrypt unregistered [ 9176.869394] Key type ._llcrypt registered [ 9176.871192] Key type .llcrypt registered [ 9176.920992] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9177.353922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9177.384714] alg: No test for adler32 (adler32-zlib) [ 9178.252769] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 9178.345173] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9179.935150] Key type lgssc registered [ 9180.351703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9183.365850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9183.377530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.585693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.114773] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9191.174771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9191.340530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9193.023404] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9195.232409] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9196.536599] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9200.612167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9200.762454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9203.014967] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_default_debug -1 all [ 9205.257647] Lustre: DEBUG MARKER: oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9217.503899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9217.506340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.510866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9220.577946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9220.578317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9220.578672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9220.591074] Lustre: Skipped 1 previous similar message [ 9222.574248] LustreError: 386352:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9222.625402] Lustre: server umount lustre-OST0000 complete [ 9225.696897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.697738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9225.702434] Lustre: Skipped 1 previous similar message [ 9230.661032] LustreError: 386606:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9230.663748] LustreError: 386606:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9230.699602] Lustre: server umount lustre-MDT0000 complete [ 9230.817346] LustreError: 385568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.823742] LustreError: 385568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9232.605185] LustreError: 383548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324493 with bad export cookie 14098543421427374592 [ 9232.608936] LustreError: MGC192.168.203.147@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9232.613208] LustreError: 383548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9232.702468] LustreError: 386855:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9232.705798] LustreError: 386855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9232.784479] Lustre: server umount lustre-MDT0001 complete [ 9238.313313] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9239.499301] Key type lgssc unregistered [ 9239.647389] LNet: 387530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9239.651559] LNetError: 387530:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9239.660736] LNet: Removed LNI 192.168.203.147@tcp [ 9240.038798] Key type .llcrypt unregistered [ 9240.040189] Key type ._llcrypt unregistered [ 9271.725454] Key type ._llcrypt registered [ 9271.726797] Key type .llcrypt registered [ 9271.767827] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9272.207267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9272.236096] alg: No test for adler32 (adler32-zlib) [ 9273.137738] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 9273.263601] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9274.879315] Key type lgssc registered [ 9275.368930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9540.244127] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9541.344648] Key type lgssc unregistered [ 9541.483435] LNet: 392616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9541.488400] LNetError: 392616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9541.499306] LNet: Removed LNI 192.168.203.147@tcp [ 9541.803391] Key type .llcrypt unregistered [ 9541.805660] Key type ._llcrypt unregistered [ 9558.902866] Key type ._llcrypt registered [ 9558.904358] Key type .llcrypt registered [ 9558.946090] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing set_hostid [ 9585.288125] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing load_modules_local [ 9585.759483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9585.766077] alg: No test for adler32 (adler32-zlib) [ 9586.625188] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 9586.722713] LNet: Added LNI 192.168.203.147@tcp [8/256/0/180] [ 9588.319119] Key type lgssc registered [ 9588.704902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9595.878498] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9600.266865] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9603.755564] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9607.453329] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9608.039673] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9148 sec ========== 22:34:29 (1767324869) [ 9608.586537] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:34:30 (1767324870) === [ 9609.776987] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:34:31 (1767324871) === [ 9620.636410] Lustre: DEBUG MARKER: oleg347-server.virtnet: executing unload_modules_local [ 9621.722147] Key type lgssc unregistered [ 9621.865503] LNet: 402622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9621.869253] LNetError: 402622:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9621.879509] LNet: Removed LNI 192.168.203.147@tcp [ 9622.204449] Key type .llcrypt unregistered [ 9622.205608] Key type ._llcrypt unregistered